Customized E-Commerce Tags in Realtime Multimedia Content

ABSTRACT

Techniques described herein are directed to receiving audio input during playback of multimedia content to trigger purchase and payment for an item represented in the multimedia content. In an example, multimedia content is sent to a user device. The multimedia content may contain a representation of an item and an indication that the item can be purchased. Based on receipt of an indication that trigger words were spoken during playback, a recording is initiated. A purchase intent of the user for the item is determined based on the recording. A graphical user interface is displayed via the user device to initiate payment by the user for the item.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of and claims priority to, U.S.patent application Ser. No. 17/014,250, filed on Sep. 8, 2020, which isincorporated by reference herein in its entirety.

TECHNICAL FIELD

The presence of multimedia content, such as videos that include a seriesof images and related audio, has become ubiquitous. While suchmultimedia content may be prerecorded, in some instances, the multimediacontent may be provided in a live or near-live fashion to a givenaudience. Such multimedia content may be utilized for various purposes,such as to inform consumers of the availability of items for purchaseand to provide details about such items.

BRIEF DESCRIPTION OF THE DRAWINGS

Features of the present disclosure, its nature and various advantages,will be more apparent upon consideration of the following detaileddescription, taken in conjunction with the accompanying drawings. Thedetailed description is set forth below with reference to theaccompanying figures. In the figures, the left-most digit(s) of areference number identifies the figure in which the reference numberfirst appears. The use of the same reference numbers in differentfigures indicates similar or identical items. The systems depicted inthe accompanying figures are not to scale and components within thefigures may be depicted not to scale with each other.

FIG. 1 illustrates an example environment for electronic commerce(“e-commerce”) tags in multimedia content and customized e-commerce tagsin real-time multimedia content.

FIG. 2 illustrates an example conceptual diagram showing the output ofmultimedia content over time and changes in displayed interactiveelements while the multimedia content is presented.

FIG. 3 illustrates an example conceptual diagram showing examples foritem recognition for items referenced in multimedia content.

FIG. 4A illustrates an example user device displaying a first exampleinteractive element overlaid on multimedia content.

FIG. 4B illustrates an example user device displaying a second exampleinteractive element overlaid on multimedia content.

FIG. 4C illustrates an example user device displaying a third exampleinteractive element overlaid on multimedia content.

FIG. 4D illustrates an example user device displaying a fourth exampleinteractive element overlaid on multimedia content.

FIG. 5 illustrates an example process for determining and utilizinginventory data in real time in association with multimedia contentrepresenting an item.

FIG. 6 illustrates an example process for determining whether toaggregate item selections during presentation of multimedia content.

FIG. 7 illustrates an example process for modifying interactive elementsbased at least in part on user preference data.

FIG. 8 illustrates an example process for utilizing user preferences formodifying display of item information and for pre-populating purchasinguser interfaces.

FIG. 9 illustrates an example process for e-commerce tags in multimediacontent.

FIG. 10 illustrates another example process for e-commerce tags inmultimedia content.

FIG. 11 illustrates an example process for customized e-commerce tags inreal-time multimedia content.

FIG. 12 illustrates a sequence diagram of a process for customizede-commerce tags in real-time multimedia content.

FIG. 13 illustrates an example merchant ecosystem for facilitating,among other things, techniques described herein.

FIG. 14 illustrates additional details associated with individualcomponents of the merchant ecosystem described above in FIG. 13 .

DETAILED DESCRIPTION

Techniques described herein are directed to, among other things, thegeneration of, association of, and/or use of e-commerce tags inmultimedia content, and more generally, improve the use of electronicdevices with the multimedia content. E-commerce tags can be selectableportions associated with multimedia content that may include e-commercespecific actions, such as purchasing actions, item information actions,and/or other shopping related actions, for example. Interactive elementsmay include any selectable element that may be presented to a user andbe configured to receive user input indicating selection. Multimediacontent may be any content that includes image data and/or audio data,for example. The multimedia content may be considered video having asequence of images, and in examples audio that accompanies the images.For example, a user (e.g., a merchant) can post an image, video, or thelike (hereinafter “content”) via a platform. Such content can depictitem(s) (e.g., goods and/or services). In some examples, the content canbe associated with an intent to sell an item depicted in the content(e.g., text associated with an image indicating that the user is lookingto sell an item depicted in the image, speech associated with a videoindicating that the user is looking to sell an item depicted in thevideo, etc.). In other examples, the content may not be associated withan intent to sell (e.g., no explicit or implicit indication that theuser desires to sell anything depicted in the content). In at least oneexample, techniques described herein alleviate the need for usersinterested in selling via certain platforms to perform any action thatthey usually wouldn't perform prior to posting content to suchplatforms. That is, users interested in selling via these platforms canpost content to one or more platforms and techniques described hereinare directed to creating selling opportunities and facilitatingtransactions based on such content. In one implementation, suchplatforms are “pure” multimedia content platforms, such as YouTubeVideos, YouTube Live, Instagram Videos, Live Instagram Feed, and so on.In such cases, the disclosed methods and systems can communicativelycouple with the content platforms to offer interactive elements ande-commerce tags. In another implementation, an e-commerce platform canprovide functionalities to publish content for sale and/or processpayments.

Take, for example, a merchant that has created or is otherwiseassociated with multimedia content that references one or more items.The multimedia content may include a visual representation of the itemsand/or speech associated with the multimedia content may audibly presenta description of the items. In examples, such multimedia content may becreated for the purpose of exposing potential customers to the itemsreferenced in the multimedia content with the hopes of those customerspurchasing the items. It should be understood that when “items” arediscussed herein, those items may include one or more goods (e.g.,physical products) and/or one or more services (e.g., provided in-personor virtual) that may be offered by a merchant. It should also beunderstood that when an item is described as being “referenced,” theitem may be visually (e.g., through gestures) shown in the multimediacontent and/or audibly discussed or otherwise referred to. Whilepresenting multimedia content to customers, such as via one or moreonline platforms, may peak interest in certain customers, if customersdesire to purchase the referenced items, they may need to navigate to awebsite or other platform that allows the customers to manually add thereferenced items to a “cart” and proceed with a typical checkout processto purchase the items. That is, customers often are required to leavecontent they are currently viewing to access another online platform(e.g., website, or the like) to view additional information, add item(s)to a cart, and/or otherwise engage in a payment flow to purchase theitem(s). This friction in the purchasing process detracts from buyerengagement with the merchant, and could potentially lead to loss of saleopportunities for the merchant. Additionally, while merchants may beadept at generating commercials and other forms of multimedia content,they may not be skilled in generating user interfaces, surfacing itemsthat may be relevant to particular consumer of the content, andformatting displayable interactive elements that ease a customer'sability to purchase the referenced items.

Techniques described herein are directed to the generation of, inexamples, on-the-fly interactive elements, e.g., e-commerce tags, andthe overlay of those interactive elements on the multimedia content whenoutput to a customer. In one implementation, such interactive elementsare generated and intelligently positioned without merchantintervention. Utilizing the techniques described herein, a merchant canpost multimedia content to a given platform, and a payment processingservice provider may perform operations that result in the generation ofitem and/or user-centric interactive elements that may be overlaid onthe multimedia content when output to the customer. For example, thepayment processing service provider may receive an instance of themultimedia content. The multimedia content may be received from a systemassociated with a given merchant and/or from one or more third-partysystems, such as systems associated with platforms such as social mediaplatforms. The payment processing service provider system may beutilized to receive the multimedia content and generate the interactiveelements as described herein.

For example, the payment processing service provider system may beconfigured to receive the multimedia content and/or to retrieve themultimedia content. For example, a merchant system or other system maypush the multimedia content to the payment processing service providersystem without a specific request for such content. In other examples,the content component may query one or more other systems for themultimedia content. In still other examples, the payment processingservice provider system may receive an indication that multimediacontent associated with a given merchant has been requested to be outputon a user device associated with a customer. In these examples, thepayment processing service provider system may query for an instance ofthe multimedia content and perform the techniques for generating aninteractive element overlay, such as prior to the multimedia contentbeing output on the user device. In this way, techniques describedherein can intelligently generate interactive element overlays based atleast in part on the multimedia content without, or with minimal, inputfrom merchants.

The payment processing service provider system may analyze themultimedia content and/or related data to identify one or more itemsreferenced in the multimedia content. For example, the paymentprocessing service provider system may utilize image data of themultimedia content to identify items depicted in the image data. Suchanalysis may include the use of computer vision techniques to identifythe presence of objects in given image data and then to identify theobjects themselves and/or object categories to which the objects belong(e.g., shirt, pants, hat, watch, etc.). Additional details on the use ofcomputer vision techniques is provided below. Additionally, oralternatively, when the multimedia content includes user speech, speechrecognition and natural language understanding techniques may beutilized to identify the speech, generate text data representing thespeech, and then determine an intent and/or purpose of the speech. Bydoing so, the payment processing service provider system may identifyitems referenced in the multimedia content as well as, in examples,attributes of the items (e.g., color, size, brand, quality (such asnew/consigned), fulfillment method, etc.). Additionally, oralternatively, the payment processing service provider system mayutilize metadata associated with the multimedia content and/or themerchant from which the content was provided to identify the items. Forexample, the merchant system may provide metadata indicating the itemsreferenced in the multimedia content. In examples, one or more otherusers may have commented on or otherwise provided information inrelation to the multimedia content. In these and other examples, some orall of this data may be utilized by the payment processing serviceprovider system to identify the items and/or attributes associated withthe items referenced in the multimedia content. That is, techniquesdescribed herein can utilize machine-trained models to intelligentlyidentify item(s) referenced in multimedia content. This can streamlinethe generation of interactive multimedia content, thereby providingaccurate interactive elements for display with minimal or no merchantinvolvement.

The payment processing service provider system may receive and/ordetermine item information associated with the items referenced in themultimedia content. For example, the merchant system may provide dataindicating information associated with the referenced items. Thatinformation may include information related to attributes of the item(e.g., sizes, colors, brands, item types, item options, etc.).Additionally, or alternatively, the item-information component may queryone or more systems for item information. By way of example, the paymentprocessing service provider system may query the merchant system forinventory data indicating a current inventory of the item(s), such as ata time that the multimedia content is being output. In examples, themerchant system may return the inventory data and that inventory datamay be utilized to inform a customer of a current inventory of items asavailable from the merchant. In other examples, such as where theinventory data indicates the item is out of stock and/or when userpreferences indicate that a different merchant is preferred by thecustomer, an indication of the current inventory of one or more othermerchants may be retrieved and displayed on the user device. The paymentprocessing service provider system may also receive item information. Inexamples where an item is no longer available, the system may beconfigured to alter the multimedia content to remove the portion of themultimedia content that references the no-longer-available item. In thisway, techniques described herein can identify item attributes specificto given multimedia content without needing human input on those itemattributes and in a way that can be used for generating specificinteractive elements.

In at least one example, operations performed by the payment processingservice provider system can leverage a multi-party merchant ecosystem.That is, in some examples, a payment processing service provider (e.g.,server(s) associated therewith) can communicate with end users (e.g.,customers and/or merchants) via respective user computing devices, andover one or more networks. Such a remote, network-connected multi-partymerchant ecosystem can enable the payment processing service provider toaccess data associated with multiple, different merchants and/orcustomers, and use such data for intelligently generating interactiveelements, in some examples, in real time or near-real time. Having apayment processing service provider, which may have access to multipledisparate merchants and multiple disparate platforms, perform theprocesses described herein allows for the unique generation and use ofmerchant-related data and multimedia content to generate interactiveelements to be displayed intelligently with multimedia content.

In examples, the identifying information about the item may be utilizedto construct a three-dimensional representation of an item that isdisplayed with respect to the multimedia content. The three-dimensionalrepresentation of the item may also be displayed to the user, such aswith an interactive element.

The payment processing service provider system may be configured toutilize the item recognition data and/or the item information data togenerate data representing an interactive element. The interactiveelement may be configured such that, when the multimedia content isoutput on the user device, the interactive element is also presented,such as in the form of an overlay. The interactive element may be, inexamples, specific to the multimedia content, the items referencedtherein, the item attributes, and/or user preferences. For example,utilizing the data received and/or determined as described herein, thepayment processing service provider system may determine a type ofinteractive element to generate. Interactive element types may include,for example, a selectable link, a quick response code (“QR code”) and/orother scannable code, an indicator that voice input for selection of theinteractive element is enabled, an indicator that gesture input forselection of the interactive element is enabled, etc. It should beunderstood that while several examples of element types have beenprovided herein, this disclosure includes any element type that allowsuser input to be received. Determining the type of interactive elementto associate with given multimedia content may be based at least in parton a device type of the user device. For example, if the device typeindicates that the device includes a camera, gesture-based interactiveelements may be utilized, or if the device type indicates the devicedoes not include a touchscreen, the interactive element may beconfigured to accept user input other than touchscreen input.Additionally, or alternatively, purchase history associated with a userprofile that is being utilized to view the multimedia content may beutilized to determine past user input types, and that information may beutilized to determine the type of interactive element to be generated.In this way, techniques described herein can generate new data on thefly that may be configured to cause user devices to change displayedcontent in a time sensitive manner.

In addition to the type of interactive element, the payment processingservice provider system may be configured to determine one or more otheraspects associated with the interactive element, such as when to displaythe interactive element with respect to the multimedia content, where todisplay the interactive element with respect to a viewable window of auser device, an amount and/or type of item detail to display, and/orfunctionality that occurs when the interactive element is selected, forexample. By way of example, the payment processing service providersystem may determine when to display the interactive element based atleast in part on data indicating when the item begins being referencedin the multimedia content and when the item ceases being referenced. Forexample, given content may be two minutes in length, but the item maynot start being referenced until the 30-second mark and then ceasesbeing referenced at the 1-minute mark. Utilizing the item-recognitiondata described herein, the payment processing service provider systemmay generate the interactive element to be configured to be displayedonly during the time frame when the item is referenced. With respect todetermining where to display the interactive element, the paymentprocessing service provider system may determine, utilizing theitem-recognition data, a relative location of the item as depicted inthe multimedia content with respect to a viewable window of the userdevice. For example, the item-recognition data may indicate a locationof the object as identified in the image data, and the interactiveelement may be generated such that, when displayed, the interactiveelement may be positioned near the object but not, in examples, over topof the object. This will allow a user to see both the object and theinteractive element, and perceive that the object and the interactiveelement are associated with each other, while the multimedia content isoutput.

With respect to determining an amount and/or type of item detail todisplay, the payment processing service provider system may utilize theitem information to determine attributes associated with the referenceditem. In some examples, all of the attributes may be included in theinteractive element. However, in other examples, only a portion of theattributes may be included. For example, utilizing the historical dataassociated with the user profile, one or more user preferences may bereceived and/or determined, and those user preferences may inform theselection of which item information to include in the interactiveelement. For example, the historical data may indicate that the userassociated with the user profile at issue purchases more items with acertain degree of item detail and/or certain types of item detailprovided. By way of additional example, the historical data may be dataassociated with more than (or other than) the user profile at issue,such as historical data associated with customers of the merchant,customers of different merchants, and/or customers in general. In otherexamples, the user preferences may be utilized to upsell or otherwisebundle items based at least in part on prior bundled purchases by theuser.

With respect to determining the functionality that will occur uponselection of the interactive element, the payment processing serviceprovider system may receive and/or determine data indicating userpreferences for selection functionality. Those user preferences mayindicate that a user desires to have a purchasing user interfacedisplayed upon selection of an interactive element. Those userpreferences may, in other examples, indicate that a user desires to havethe purchasing user interface displayed only after cessation of themultimedia content or otherwise at some time after selection of a giveninteractive element. This functionality may allow the user to selectmultiple interactive elements corresponding each to different itemsbefore being presented with a purchasing user interface. In theseexamples, the interactive elements may be configured to be selected andthen data indicating those selections may be saved until cessation ofthe multimedia content. Additionally, the user preference informationdescribed herein may be utilized to recommend additional multimediacontent to display to the user and/or to influence how users navigatebetween multimedia content.

The payment processing service provider system may be configured togenerate commands that, among other things, cause devices such as userdevices to perform actions. For example, the payment processing serviceprovider system may generate a command to cause the interactive elementto be presented along with the multimedia content. The paymentprocessing service provider system may also generate a command to causethe user device to display a purchasing user interface in response toselection of one or more interactive elements. The payment processingservice provider system may also generate a command to cause the userdevice to display information in the user interface. For example, one ormore user-input fields of the purchasing user interface may beprepopulated based at least in part on some or all of the data discussedherein. For example, data from the user profile may be utilized toprepopulate attributes and/or options associated with a selected item.Additionally, or alternatively, the item information determined from themultimedia content may be utilized to prepopulate the item attributes.Additionally, payment information from past transactions associated withthe user profile may be utilized to prepopulate payment options,delivery addresses, etc. on the purchasing user interface. In this way,upon selection of the interactive element(s), the purchasing userinterface may be automatically displayed and prepopulated with item andpayment instrument information such that a user may need to only confirmthe purchase without providing any additional input to acquire the item.It should be understood that when the purchasing user interface isdisplayed, this user interface may be associated with the merchant, suchas a merchant website that would allow for purchasing, and/or the userinterface may be associated with the payment processing serviceprovider, such as a web-based and/or application-based user interfacethat would allow for purchasing. In this way, techniques describedherein can cause applications of user devices to activate and causedisplay of user interfaces with time-sensitive, secure data prepopulatedin those interfaces for use by the user.

The payment processing service provider system may be configured toreceive feedback data associated with the interactive elements. Thefeedback data may indicate positive and/or negative comments about theinteractive elements, display of the interactive elements, and/orfunctionality associated with selection of the interactive elements.This feedback data may be collected and utilized to improve thegeneration of interactive elements and/or related functionality. Forexample, the feedback data may be formatted as input data (or otherwisetraining data) to one or more machine learning models. This input may beutilized to train the machine learning models, which may be utilized bythe various components of the systems described herein to perform one ormore of the operations described with respect to those systems. In thisway, techniques described herein can utilize data provided by users orotherwise commenters for generating new, more accurate interactiveelements.

It should be understood that the operations described herein may beperformed on prerecorded multimedia content and/or on live and/ornear-live streaming content. When live and/or near-live streamingcontent is utilized, the content may be received at the paymentprocessing service provider system and may be utilized for itemrecognition and interactive element generation. This may be performedbefore the multimedia content is sent to the user device and/or whilethe user device is outputting the multimedia content. With respect tothe live and/or near-live examples, the platform associated with thestreaming content may be any platform that allows for customerinteraction, such as conference calling platforms, social mediaplatforms, etc. In these examples, functionality associated with theinteractive elements may be in the form of a downloadable widget orotherwise an application that may allow for interactive elements to bedisplayed.

It should be noted that the exchange of data and/or information asdescribed herein may be performed only in situations where a user hasprovided consent for the exchange of such information. For example, uponsetup of devices and/or initiation of applications, a user may beprovided with the opportunity to opt in and/or opt out of data exchangesbetween devices and/or for performance of the functionalities describedherein. Additionally, when one of the devices is associated with a firstuser account and another of the devices is associated with a second useraccount, user consent may be obtained before performing some, any, orall of the operations and/or processes described herein. Additionally,the operations performed by the components of the systems describedherein may be performed only in situations where a user has providedconsent for performance of the operations.

The embodiments described herein relate to the generation of new“embedded data,” such as links which may be generated on-the-fly in someembodiments, that is used to display selectable links, quick referencecodes, tags, etc. while multimedia content is being output on a screenof a user device. Additionally, the dynamic display of purchasinginformation (such as sizes, colors, inventory, price, etc.) both duringvideo output and after a user clicks, scans, or otherwise selects thelink are computer-centric solutions that utilize disparate data sourcesto produce anew and useful result specific to computing environments.Further, the techniques described herein include the generation of newdata, such as the link data described herein which is specific to givenmultimedia content and is configured to be displayed, in examples,differently for different user devices, for example customized based onuser profiles. By doing so, online platform marketplaces may beaugmented such that information displayed to potential customers is moreaccurate, tailored to the specific customer, is presented in a timesensitive manner, and provides functionality that could only exist in acomputer-centric environment. Commands to display selectable elements asdescribed herein may also be configured to cause applications on userdevices to initiate and/or activate and cause display of time-sensitiveinformation, such as without user input. By so doing, a material changein the user devices themselves is achieved such that the user devicesmay perform functions that otherwise would not have been possible absentthe computer-centric solutions described herein. Additionally, given theincredible amount of information on all items available for sale, evenwith respect to a single merchant, and the possible item options andpayment options, the innovations described herein may be utilized tofilter internet content such that only relevant selectable links aredisplayed (and only in specific screen locations and for specific timesin given examples) and only relevant item and purchasing information isprovided when a selectable element is displayed. By so doing, thepresent methods and systems provide filtering internet content in amanner that is new and increases the functionality of user devices andmerchant devices alike.

The embodiments described herein use information about the multimediacontent similarities including but not limited to spatial and temporalsimilarities in the multimedia content; specifically, collecting andusing subject-specific information about the speaker that is featured inthe multimedia, e.g., video. In one embodiment, the disclosed methodsand systems leverage the fact that usually one participant of a videoconference speaks at a time and that the video displayed in a mainwindow generally focusses on the speaker and not on the othernon-speaking participants (whose videos may also be displayed buttypically in smaller sidebar windows and not in the main window),filtering video information according to one embodiment is performed ina participant-specific manner. The participant-specific video filteringis based at least on the manner in which a particular speaker speaks,e.g., the speaker's facial expressions, movement of the head and eyes,hand gestures, etc., generally indicate or otherwise point to iteminformation. In various examples described below, suchparticipant-specific knowledge is learned via machine learning, and thatknowledge is used to minimize redundancies in video transmission inaddition to (or instead of) spatial and temporal redundancies. Thisallows for the focus to be on one speaker and or related items asopposed to other speakers and other items in the video. This also allowsfor a higher compression ratio than that achieved using standardizedvideo filtering and recognition schemes. The learnedparticipant-specific knowledge is used in decoding or reconstructing theencoded video as well, so that a high or desired perception quality canbe maintained despite the higher compression ratio. Since thecompression ratio is generally higher relative to standardized encoding,the frame rate of video transmission need not be lowered when theavailable bandwidth is low. Thus, a high or desired perception qualitycan be maintained in video conferencing, where buffering to accommodatevariable frame transmission rates is not available. More specifically,such techniques allow more accurate tagging and identification ofrelevant items for commerce.

Examples of multimedia content include multimedia transmissions, whichare broadly classified into two categories, namely, stored videotransmission and live streaming. Stored video transmission can be usedto transmit videos that are produced at one time and are transmitted andviewed at a later time. Examples of such videos include movies,previously recorded TV shows, previously recorded sports events, etc. Inlive streaming, on the other hand, production, transmission, reception,and display of the video occur in real time. Examples of live streaminginclude video chats, video conferences, live transmission of news, liveshopping events, live social media feeds, live infomercials, etc. Onemain difference between stored video transmission and live streaming isthe availability of buffering in the former, and lack thereof in thelatter. The embodiments described herein provide generation ofe-commerce tags in any kind of multimedia transmission. In case ofstored or previously buffered videos, the disclosed methods and systemscan analyze and then generate the tags to be associated with relevantsections of the media. In case of live multimedia, the methods andsystems can analyze the content in real-time or near-real time andgenerate tags on the fly with the most current information regarding theitems.

The present disclosure provides an overall understanding of theprinciples of the structure, function, manufacture, and use of thesystems and methods disclosed herein. One or more examples of thepresent disclosure are illustrated in the accompanying drawings. Thoseof ordinary skill in the art will understand that the systems andmethods specifically described herein and illustrated in theaccompanying drawings are non-limiting embodiments. The featuresillustrated or described in connection with one embodiment may becombined with the features of other embodiments, including as betweensystems and methods. Such modifications and variations are intended tobe included within the scope of the appended claims.

Additional details are described below with reference to several exampleembodiments.

FIG. 1 illustrates an example environment for electronic commerce(“e-commerce”) tags in multimedia content and customized e-commerce tagsin real-time multimedia content. In FIG. 1 , server(s) 104 can beassociated with a payment processing service provider, which cancommunicate with user computing devices, such as a merchant device 106(also described herein as a merchant device and/or a merchant system)and a buyer device 102 (also described herein as a user device), vianetwork(s) 108. That is, the merchant device 106 and the buyer device102 are network-connected devices that enable end users (e.g., amerchant and a buyer, respectively) to access services provided by thepayment processing service provider (e.g., via the server(s) 104).Additional details associated with the server(s) 104, the user computingdevices (e.g., 102, 106), and the network(s) 108 are described belowwith reference to FIGS. 13 and 14 .

In at least one example, the server(s) 102 can include a paymentprocessing component 152. The payment processing component 152 can,among other things, process transactions. That is, in at least oneexample, the payment processing component 152 can access payment dataassociated with a user, send a request for authorization of the paymentdata to a payment service provider, and process a transaction based on aresponse from the payment service provider. In other examples, thepayment processing component 152 can access an account maintained by thepayment processing service provider and can use funds associated withthe account to process a transaction. Additional details associated withthe payment processing component 152 are described below.

In at least one example, the payment processing service provider canexpose functionality and/or services via one or more APIs 148, therebyenabling functionality and/or services described herein to be integratedinto various functional components of the environment 100. The API(s)148, which can be associated with the server(s) 104, can exposefunctionality described herein and/or avail payment processing servicesto various functional components associated with the environment 100. Atleast one of the API(s) 148 can be a private API, thereby availingservices and/or functionalities to functional components (e.g.,applications, etc.) that are developed internally (e.g., by developersassociated with the payment processing service provider). At least oneof the API(s) 148 can be an open or public API, which is a publiclyavailable API that provides third-party developers (e.g., social mediaservice providers described herein) with programmatic access to aproprietary software application or web service of the paymentprocessing service provider. That is, the open or public API(s) canenable functionality and/or services of the payment processing serviceprovider to be integrated into multimedia content platforms. The API(s)148 can include sets of requirements that govern how applications, orother functional components, can interact with one another.

In some examples, the payment processing service provider can providethird-party entities with a software developer kit (“SDK”) that mayutilize functionality exposed by the API(s) 148. The SDK can includesoftware development tools that allow a third-party developer (i.e., adeveloper that is separate from the payment processing service provider)to include functionality and/or avail services as descried herein. TheSDK and/or the API(s) 148 may include one or more libraries, programmingcode, executables, other utilities, and documentation that allows adeveloper to directly include functionality and/or avail servicesdescribed herein within an application, such as third-party applicationsproviding social networking services, as described herein.

In at least one example, the server(s) 104 can include, or otherwisehave access to, data store(s) 150. The data store(s) 150 can store,among other types of data, user profiles and inventory records.Additionally, the server(s) 104 can include a user registry 146, whichmay also include user profiles and/or include associations between userprofiles and merchant profiles. For instance, a user profile of thebuyer can store payment data associated with payment instrument(s) ofthe buyer. In some examples, an account maintained by the paymentprocessing service provider on behalf of the buyer can be mapped to, orotherwise associated with, the user profile of the buyer. Such anaccount can store funds received from peer-to-peer payment transactions,deposits from employers, transfers from other accounts of the buyer, andso on. Additionally or alternatively, a user profile of the merchant canbe mapped to, or otherwise associated with, an account of the merchant(which can be maintained by the payment processing service provider, abank, or another payment service). Additional details are providedbelow.

As illustrated in FIG. 1 , the buyer device 102 is associated with userinterface(s) 122 that enable the buyer to interact with the buyer device102. The user interface(s) 122 can be presented via web browsers,applications (e.g., desktop or otherwise dedicated, provided by thepayment processing provider, provided by a third-party, etc.), or thelike to enable the buyer to access functionality and/or services asdescribed herein. Similarly, the merchant device 106 can be associatedwith user interface(s) which can be presented via web browsers,applications (e.g., desktop or otherwise dedicated, provided by thepayment processing provider, provided by a third-party, etc.), or thelike to enable the merchant to interact with the merchant device 106 andaccess functionality and/or services as described herein.

In at least one example, a user interface of the user interface(s) 122can be presented via a multimedia platform (e.g., website, application,etc.) associated with a provider of multimedia content. Functionalityand/or services of the payment processing service provider can beintegrated into the social media platform via the API(s) 148 and/orSDKs. In at least one example, the merchant can post content via theplatform. In FIG. 1 , the content is multimedia content, but inadditional or alternative examples, the content can be any other type ofcontent. In at least one example, the buyer can access and/or consumethe content via a user interface of the user interface(s) 122 that ispresented via the platform. That is, the merchant and the buyer can eachaccess the platform via user interfaces presented via their respectivedevices.

In at least one example, one or more users can respond to content, forexample, via comments (which can include text, images, emojis, etc.),interactions with a button or other actuation mechanism (e.g., like,dislike, funny, love, etc.), and so on. Such responses can be posted innear real-time. For instance, one or more users can respond to themultimedia content posed by the merchant.

As illustrated above, the environment 100 may include the buyer device102, the server(s) 104, and/or the merchant device 106. The buyer device102, in addition to the components discussed above, may include one ormore components such as one or more processors 110, one or more networkinterfaces 112, memory 114, one or more microphones 116, one or morespeakers 118, and/or one or more displays 120. The microphones 116 maybe configured to receive audio from the environment 100 may generatecorresponding audio data, which may be utilized as discussed herein. Thespeakers 118 may be configured to output audio, such as audiocorresponding to at least a portion of the multimedia content output bythe buyer device 102. The displays 120 may be configured to presentimages (which may be described as video) corresponding to at least aportion of the multimedia content output by the buyer device 102. Thememory 114 may include one or more components such as the userinterface(s) 122 (discussed above), and one or more applications 124.The applications 124 may be associated with content providers,merchants, and/or the payment processing service provider. The merchantdevice 106 may include the same or similar components that may performthe same or similar functions. It should be noted that the merchantdevice 106, like the other devices and systems described herein, maytake one or more forms, such as a computing device, a laptop computer, aphone, and/or components thereof, for example.

The server(s) 104 may include one or more components including, forexample, one or more processors 126, one or more network interfaces 128,and/or memory 130. The memory 130 may include one or more componentssuch as, for example, a content component 132, an item-recognitioncomponent 134, an item-information component 136, an interactive-elementgenerator 138, a command generator 140, a feedback component 142, one ormore machine learning models 144, a user registry 146, one or more APIs148, one or more datastores 150, and/or the payment processing component152. The user registry 146, APIs 148, datastores 150, and paymentprocessing component 152 have been described above. The other componentswill be described below by way of example.

For example, the content component 132 may be configured to receive themultimedia content and/or to retrieve the multimedia content. Forexample, the merchant device 106 or other system may push the multimediacontent to the payment processing service provider without a specificrequest for such content. In other examples, the content component 132may query one or more other systems for the multimedia content. In stillother examples, the content component 132 may receive an indication thatmultimedia content associated with a given merchant has been requestedto be output on a buyer device 102 associated with a customer. In theseexamples, the content component 132 may query for an instance of themultimedia content and perform the techniques for generating aninteractive element overlay, such as prior to the multimedia contentbeing output on the buyer device 102.

The item-recognition component 134 may analyze the multimedia contentand/or related data to identify one or more items referenced in themultimedia content. For example, the item-recognition component 134 mayutilize image data of the multimedia content to identify items depictedin the image data. Such analysis may include the use of computer visiontechniques to identify the presence of objects in given image data andthen to identify the objects themselves and/or object categories towhich the objects belong (e.g., shirt, pants, hat, watch, etc.).Additionally, or alternatively, when the multimedia content includesuser speech, speech recognition and natural language understandingtechniques may be utilized to identify the speech, generate text datarepresenting the speech, and then determine an intent and/or purpose ofthe speech. By doing so, the item-recognition component 134 may identifyitems referenced in the multimedia content as well as, in examples,attributes of the items (e.g., color, size, brand, etc.). Additionally,or alternatively, the item-recognition component 134 may utilizemetadata associated with the multimedia content and/or the merchant fromwhich the content was provided to identify the items. For example, themerchant device 106 may provide metadata indicating the items referencedin the multimedia content. In examples, one or more other users may havecommented on or otherwise provided information in relation to themultimedia content. In these and other examples, some or all of thisdata may be utilized by the item-recognition component 134 to identifythe items and/or attributes associated with the items referenced in themultimedia content. It should be understood that when voice recognitionis described herein as being usable to identify items, voice recognitionmay be utilized, in examples, along with image recognition as describedherein for item recognition.

With respect to computer vision techniques, computer vision includesmethods for acquiring, processing, analyzing, and understanding imagesand, in general, high-dimensional data from the real world in order toproduce numerical or symbolic information, e.g., in the form ofdecisions. Computer vision seeks to duplicate the abilities of humanvision by electronically perceiving and understanding an image.Understanding in this context means the transformation of visual images(the input of the retina) into descriptions of the world that caninterface with other thought processes and elicit appropriate action.This image understanding can be seen as the disentangling of symbolicinformation from image data using models constructed with the aid ofgeometry, physics, statistics, and learning theory. Computer vision hasalso been described as the enterprise of automating and integrating awide range of processes and representations for vision perception. As ascientific discipline, computer vision is concerned with the theorybehind artificial systems that extract information from images. Theimage data can take many forms, such as video sequences, views frommultiple cameras, or multi-dimensional data from a scanner. As atechnological discipline, computer vision seeks to apply its theoriesand models for the construction of computer vision systems.

One aspect of computer vision comprises determining whether or not theimage data contains some specific object, feature, or activity.Different varieties of computer vision recognition include: ObjectRecognition (also called object classification)—One or severalpre-specified or learned objects or object classes can be recognized,usually together with their 2D positions in the image or 3D poses in thescene. Identification—An individual instance of an object is recognized.Examples include identification of a specific person's face orfingerprint, identification of handwritten digits, or identification ofa specific vehicle. Detection—The image data are scanned for a specificcondition. Examples include detection of possible abnormal cells ortissues in medical images or detection of a vehicle in an automatic roadtoll system. Detection based on relatively simple and fast computationsis sometimes used for finding smaller regions of interesting image datathat can be further analyzed by more computationally demandingtechniques to produce a correct interpretation.

Several specialized tasks based on computer vision recognition exist,such as: Optical Character Recognition (OCR)—Identifying characters inimages of printed or handwritten text, usually with a view to encodingthe text in a format more amenable to editing or indexing (e.g., ASCII).2D Code Reading—Reading of 2D codes such as data matrix and QR codes.Facial Recognition. Shape Recognition Technology (SRT)—Differentiatinghuman beings (e.g., head and shoulder patterns) from objects.

Some functions and components (e.g., hardware) found in many computervision systems are described here. For example, a digital image isproduced by one or several image sensors, which, besides various typesof light-sensitive cameras, may include range sensors, tomographydevices, radar, ultra-sonic cameras, etc. Depending on the type ofsensor, the resulting image data may be a 2D image, a 3D volume, or animage sequence. The pixel values may correspond to light intensity inone or several spectral bands (gray images or color images), but canalso be related to various physical measures, such as depth, absorptionor reflectance of sonic or electromagnetic waves, or nuclear magneticresonance. Before a computer vision method can be applied to image datain order to extract some specific piece of information, it is usuallybeneficial to process the data in order to assure that it satisfiescertain assumptions implied by the method. Examples of pre-processinginclude, but are not limited to re-sampling in order to assure that theimage coordinate system is correct, noise reduction in order to assurethat sensor noise does not introduce false information, contrastenhancement to assure that relevant information can be detected, andscale space representation to enhance image structures at locallyappropriate scales. Image features at various levels of complexity areextracted from the image data. Typical examples of such features are:Lines, edges, and ridges; Localized interest points such as corners,blobs, or points; More complex features may be related to texture,shape, or motion. At some point in the processing a decision may be madeabout which image points or regions of the image are relevant forfurther processing. Examples are: Selection of a specific set ofinterest points; Segmentation of one or multiple image regions thatcontain a specific object of interest; Segmentation of the image intonested scene architecture comprising foreground, object groups, singleobjects, or salient object parts (also referred to as spatial-taxonscene hierarchy). At this point, the input may be a small set of data,for example a set of points or an image region that is assumed tocontain a specific object. The remaining processing may comprise, forexample: verification that the data satisfy model-based andapplication-specific assumptions; estimation of application-specificparameters, such as object pose or object size; classifying a detectedobject into different categories; and comparing and combining twodifferent views of the same object. Making the final decision requiredfor the application, for example match/no-match in recognitionapplications, may then be performed.

The item-information component 136 may receive and/or determine iteminformation associated with the items referenced in the multimediacontent. For example, the merchant device 106 may provide dataindicating information associated with the referenced items. Thatinformation may include information related to attributes of the item(e.g., sizes, colors, brands, item types, item options, etc.).Additionally, or alternatively, the item-information component 136 mayquery one or more systems for item information. By way of example, theitem-information component 136 may query the merchant device 106 forinventory data indicating a current inventory of the item(s), such as ata time that the multimedia content is being output.

The inventory data may indicate one or more attributes and/or conditionsof items. For example, the inventory data may include a stock count orother indication of a number of the item in question that is currentlyin stock. The inventory data may also include information about itemsthat are currently in stock, such as item color, item size, item type, aphysical location and/or location association of the item, and/or theavailability of the item. In examples, the merchant device 106 mayreturn the inventory data and that inventory data may be utilized toinform a customer of a current inventory of items as available from themerchant. In other examples, such as where the inventory data indicatesthe item is out of stock and/or when user preferences indicate that adifferent merchant is preferred by the customer, an indication of thecurrent inventory of one or more other merchants may be retrieved anddisplayed on the buyer device 102. The item-information component 136may also receive item information from the item-recognition component136, such as when the item-recognition component 136 determines one ormore attributes of the item utilizing the techniques described herein.When determining item inventory, machine learning models, such as thosedescribed herein, may be utilized to determine what items are likely tosell or not sell, and when the inventory amount is displayed asdescribed herein, such as part of the interactive element, the inventoryamount may be a predicted inventory based at least in part on the outputof the machine learning models.

The interactive-element generator 138 may be configured to utilize thedata received and/or determined by the item-recognition component 134and/or the item-information component 136 to generate data representingan interactive element. Interactive elements may include any selectableelement that may be presented to a user and be configured to receiveuser input indicating selection. E-commerce tags can be a type ofinteractive element and may be selectable portions associated withmultimedia content that may include e-commerce specific actions, such aspurchasing actions, item information actions, and/or other shoppingrelated actions, for example. Multimedia content may be any content thatincludes image data and/or audio data, for example. The interactiveelement may be configured such that, when the multimedia content isoutput on the buyer device 102, the interactive element is alsopresented, such as in the form of an overlay. The interactive elementmay be, in examples, specific to the multimedia content, the itemsreferenced therein, the item attributes, and/or user preferences. Forexample, utilizing the data received and/or determined as describedherein, the interactive-element generator 138 may determine a type ofinteractive element to generate. Interactive element types may include,for example, a selectable link, a quick response code (“QR code”), abarcode or other scannable element, an indicator that voice input forselection of the interactive element is enabled, an indicator thatgesture input for selection of the interactive element is enabled, etc.It should be understood that while several examples of element typeshave been provided herein, this disclosure includes any element typethat allows user input to be received. Determining the type ofinteractive element to associate with given multimedia content may bebased at least in part on a device type of the buyer device 102. Forexample, if the device type indicates that the device includes a camera,gesture-based interactive elements may be utilized, or if the devicetype indicates the device does not include a touchscreen, theinteractive element may be configured to accept user input other thantouchscreen input. Additionally, or alternatively, purchase historyassociated with a user profile that is being utilized to view themultimedia content may be utilized to determine past user input types,and that information may be utilized to determine the type ofinteractive element to be generated.

In addition to the type of interactive element, the interactive-elementgenerator 138 may be configured to determine one or more other aspectsassociated with the interactive element, such as when to display theinteractive element with respect to the multimedia content, where todisplay the interactive element with respect to a viewable window of abuyer device 102, an amount and/or type of item detail to display,and/or functionality that occurs when the interactive element isselected, for example. By way of example, the interactive-elementgenerator 138 may determine when to display the interactive elementbased at least in part on data indicating when the item begins beingreferenced in the multimedia content and when the item ceases beingreferenced. For example, given content may be two minutes in length, butthe item may not start being referenced until the 30-second mark andthen ceases being referenced at the 1-minute mark. Utilizing theitem-recognition data described herein, the interactive-elementgenerator 138 may generate the interactive element to be configured tobe displayed only during the time frame when the item is referenced.With respect to determining where to display the interactive element,the interactive-element generator 138 may determine, utilizing theitem-recognition data, a relative location of the item as depicted inthe multimedia content with respect to a viewable window of the buyerdevice 102. For example, the item-recognition data may indicate alocation of the object as identified in the image data, and theinteractive element may be generated such that, when displayed, theinteractive element may be positioned near the object but not, inexamples, over top of the object. This will allow a user to see both theobject and the interactive element, and perceive that the object and theinteractive element are associated with each other, while the multimediacontent is output.

With respect to determining an amount and/or type of item detail todisplay, the interactive-element generator 138 may utilize the iteminformation from the item-information component 136 to determineattributes associated with the referenced item. In some examples, all ofthe attributes may be included in the interactive element. However, inother examples, only a portion of the attributes may be included. Forexample, utilizing the historical data associated with the user profile,one or more user preferences may be received and/or determined, andthose user preferences may inform the selection of which iteminformation to include in the interactive element. For example, thehistorical data may indicate that the user associated with the userprofile at issue purchases more items with a certain degree of itemdetail and/or certain types of item detail provided. By way ofadditional example, the historical data may be data associated with morethan (or other than) the user profile at issue, such as historical dataassociated with customers of the merchant, customers of differentmerchants, and/or customers in general. Additionally, in examples wheredevice functionality will allow for display of augmented reality and/orvirtual reality representations of the multimedia data, the interactiveelements may be configured to also be displayed in virtual realityand/or in an augmented reality setting. This may allow for differentorientations and/or views of the interactive element, such as whenorientations and/or views of the items in the multimedia content change.

With respect to determining the functionality that will occur uponselection of the interactive element, the interactive-element generator138 may receive and/or determine data indicating user preferences forselection functionality. Those user preferences may indicate that a userdesires to have a purchasing user interface 122 displayed upon selectionof an interactive element. Those user preferences may, in otherexamples, indicate that a user desires to have the purchasing userinterface 122 displayed only after cessation of the multimedia contentor otherwise at some time after selection of a given interactiveelement. This situation may allow the user to select multipleinteractive elements corresponding each to different items before beingpresented with a purchasing user interface 122. In these examples, theinteractive elements may be configured to be selected and then dataindicating those selections may be saved until cessation of themultimedia content.

In the examples provided above, the interactive-element generator 138 isdescribed as generating specific interactive elements for specific itemsreferenced in multimedia content. In other examples, theinteractive-element generator 138 may generate a default and/or genericinteractive element that indicates an item is available but does notprovide specific details about the item. When the generic interactiveelement is selected, the payment processing service provider system mayresolve what is known about the item and/or the user profile associatedwith the user to utilize more specific information for prepopulating apurchasing user interface.

The command generator 140 may be configured to generate commands that,among other things, cause devices such as user devices to performactions. For example, the command generator 140 may generate a commandto cause the interactive element to be presented along with themultimedia content. The command generator 140 may also generate acommand to cause the buyer device 102 to display the purchasing userinterface 122 in response to selection of one or more interactiveelements. The command generator 140 may also generate a command to causethe buyer device 102 to display information in the user interface 102.For example, one or more user-input fields of the purchasing userinterface 122 may be prepopulated based at least in part on some or allof the data discussed herein. For example, data from the user profilemay be utilized to prepopulate attributes and/or options associated witha selected item. Additionally, or alternatively, the item informationdetermined from the multimedia content may be utilized to prepopulatethe item attributes. Additionally, payment information from pasttransactions associated with the user profile may be utilized toprepopulate payment options, delivery addresses, etc. on the purchasinguser interface 122. In this way, upon selection of the interactiveelement(s), the purchasing user interface 122 may be automaticallydisplayed and prepopulated with item and payment instrument informationsuch that a user may need to only confirm the purchase without providingany additional input to acquire the item. In addition to the avoid, theuser may select a “save for later” functionality that allows theinformation associated with the purchasing user interface to be saved sothe user may utilize that information for purchasing the item(s) at alater time. In other examples, instead of a purchasing user interface,selection of the interactive element may allow the user to enter into anauction for the item where bidding may occur. This may allow formultiple users to bid on the same item at the same time. As describedabove, selection of the interactive element may cause the purchasinguser interface 122 to be displayed. In examples, the purchasing userinterface 122 may be associated with the merchant. However, in otherexamples, at least a portion of the purchasing functionality may beprovided by the payment processing service provider system 104. In theseexamples, the payment processing service provider system 104 may beassociated with an application, that may be stored and/or be accessibleon the customer's devices, to allow for purchasing of the item(s)utilizing the application. In other examples, selection of interactiveelements may allow for users to engage with items in a video and, whenthe interactive elements are selected, a request may be sent to specificmerchants with those associated items to either engage directly with thecustomer, or allows the customer to directly buy the most relevant item.In these examples, interactive elements may be associated with certainportions of the multimedia content (such as portions where informationabout described items in the multimedia content is available), while theother portions are not interactive and therefore do not have any“buyable” items associated therewith. Additionally, user preferenceinformation described herein may be utilized to recommend additionalmultimedia content to display to the user and/or to influence how usersnavigate between multimedia content.

In examples, a virtual cart and/or cart data structure may be generatedbased at least in part on selection of interactive elements and/or otherinteraction by a customer with the purchasing user interface. In theseexamples, the virtual cart may include details about the selectedinteractive element, item(s) associated with the interactive element,payment and/or cost information associated with the item(s), and/orother data associated with the item(s) and/or a payment transaction forthe item(s). The cart data structure may be configured to generateand/or make available payment links and/or to make subsequentinteraction by the user easier and/or more efficient. For example, thecart data structure may be saved or otherwise stored such that the usermay return to the virtual cart at a later time to continue purchasingthe item(s) without having to select the items again and/or withouthaving to input information that has already been input and/orprepopulated. Messages may also be sent to device and/or user profilesassociated with the user utilizing the cart data structure. Thesemessages may represent reminders of the item(s) in the virtual cartand/or requests to complete purchases. In examples, the virtual cart mayinclude a fully defined item or a suggestion and/or item class that maybe refined at a later time. As such, the virtual cart may act as a“bookmark” and/or may otherwise save data associated with the item(s)and/or the user's interaction with the items via the multimedia content,interactive elements, applications, or otherwise.

Additionally, the models 144 may be trained based on transactioninformation and/or other information associated with paymenttransactions. For example, transaction information of transactionsperformed between multiple merchants and multiple buyers may be receivedat the payment processing service provider system 104. This informationmay be received from merchant computing devices associated with themerchants. In these examples, the merchant computing devices may haverespective instances of a merchant application installed thereon forconfiguring the merchant computing devise as point-of-sale (POS)terminals, respectively. The respective instances of the merchantapplication may configure the POS terminals to communicate thetransaction information over one or more networks to the paymentprocessing service provider system 104. In some examples, the POSterminals could be online terminals. Utilizing the transactioninformation, the payment processing service provider system 104 maygenerate profiles, using a model trained using at least one of merchantinformation, buyer information, and/or the transaction information.

In some implementations, the methods and systems described herein can beintegrated with voice services (e.g. Amazon's ALEXA®, Apple's SIRI®, orMicrosoft's CORTANA®) through specific API calls to such services fromwithin the multimedia interface. The present methods and systems canintegrate with the “wake words” for invoking their respective voiceservice, ecommerce and fulfillment channels. For example, speakerrecognition techniques may be utilized to determine user profilesassociated with users that provide user utterances to user devices forperforming one or more of the operations described herein. Thedetermined user profile may be utilized to customize the interactiveelements described herein, functionality that occurs when an interactiveelement is selected, and/or the information that is displayed in thepurchasing user interface. In examples, geographic location of theviewing device may also be utilized to customize the interactiveelements and/or the availability of the multimedia content, interactiveelements, and/or items. The voice interface may also be utilized toprovide the user inputs described herein. In addition, the voiceinterface may be associated with an online platform, and that platformmay be utilized to fulfill item purchasing.

In examples, a merchant dashboard may be utilized to allow the merchantto view interactions with the multimedia content and/or data associatedwith the interactions. The merchant may be able to provide input fordynamic pricing of the items based at least in part on the interactions.Additional functionality may include on the fly question answering,incentive provision, and other functionality that allows the merchant tointeract with customers. Additionally, the merchant dashboard may allowa merchant to interact live with customers and process payments withdifferent users.

The feedback component 142 may be configured to receive feedback dataassociated with the interactive elements. The feedback data may indicatepositive and/or negative comments about the interactive elements,display of the interactive elements, and/or functionality associatedwith selection of the interactive elements. This feedback data may becollected and utilized to improve the generation of interactive elementsand/or related functionality. For example, the feedback data may beformatted as input data (or otherwise training data) to the one or moremachine learning models 144. This input may be utilized to train themachine learning models 144, which may be utilized by the variouscomponents of the systems described herein to perform one or more of theoperations described with respect to those systems.

The embodiments described herein use information about the multimediacontent similarities including but not limited to spatial and temporalsimilarities in the multimedia content; specifically, collecting andusing subject-specific information about the speaker that is featured inthe multimedia, e.g., video. In one embodiment, the disclosed methodsand systems leverage the fact that usually one participant of a videoconference speaks at a time and that the video displayed in a mainwindow generally focusses on the speaker and not on the othernon-speaking participants (whose videos may also be displayed buttypically in smaller sidebar windows and not in the main window),filtering video information according to one embodiment is performed ina participant-specific manner. The participant-specific video filteringis based at least on the manner in which a particular speaker speaks,e.g., the speaker's facial expressions, movement of the head and eyes,hand gestures, etc., generally indicate or otherwise point to iteminformation. In various examples described below, suchparticipant-specific knowledge is learned via machine learning, and thatknowledge is used to minimize redundancies in video transmission inaddition to (or instead of) spatial and temporal redundancies. Thisallows for the focus to be on one speaker and or related items asopposed to other speakers and other items in the video. This also allowsfor a higher compression ratio than that achieved using standardizedvideo filtering and recognition schemes. The learnedparticipant-specific knowledge is used in decoding or reconstructing theencoded video as well, so that a high or desired perception quality canbe maintained despite the higher compression ratio. Since thecompression ratio is generally higher relative to standardized encoding,the frame rate of video transmission need not be lowered when theavailable bandwidth is low. Thus, a high or desired perception qualitycan be maintained in video conferencing, where buffering to accommodatevariable frame transmission rates is not available. More specifically,such techniques allow more accurate tagging and identification ofrelevant items for commerce.

FIG. 2 illustrates an example conceptual diagram showing the output ofmultimedia content over time and changes in displayed interactiveelements while the multimedia content is presented. The environment 200of FIG. 2 may include a buyer device 102 that is being caused to displaymultimedia content as well as interactive elements displayed as anoverlay on the multimedia content.

Specifically with respect to FIG. 2 , the multimedia content mayreference a first item 206, herein depicted as a shirt. The first item206 may be identified utilizing the one or more techniques describedelsewhere herein. The buyer device 102 may also display or otherwise beassociated with a time bar 202 and a progression element 204. The timebar 202 may provide an indication of an overall length of the multimediacontent, such as in minutes and seconds, for example. The progressionelement 204 may provide an indication of where the currently-displayedcontent is with respect to the overall length of the multimedia content.As shown in step 1 of FIG. 2 , the progression element 204 indicatesthat the first item 206 is displayed early on in the multimedia content.Also as shown in step 1, an interactive-element generator has generateda first interactive element 208 to be displayed in association with thefirst item 206. The first interactive element 208 may be generated andcustomized utilizing the techniques described elsewhere in thisdisclosure, for example with reference to FIG. 1 . As shown in step 1 ofFIG. 2 , the first interactive element 208 has been generated and causedto be displayed at a given location with respect to an interactivewindow of the buyer device 102, and the first interactive element 208includes certain item details. Interactive elements may include anyselectable element that may be presented to a user and be configured toreceive user input indicating selection. E-commerce tags can be a typeof interactive element and may be selectable portions associated withmultimedia content that may include e-commerce specific actions, such aspurchasing actions, item information actions, and/or other shoppingrelated actions, for example.

As shown in FIG. 2 , the item details include the item type (e.g.,“shirt”), a brand (e.g., “Brand A”), and an inventory count (e.g., “5”)for the item. It should be understood that any other item informationmay also or alternatively be displayed as part of the interactiveelement. Additionally or alternatively the displayed information can becustomized according to real time or near real time contextualinformation, such as time, location, viewing audience, inventory status,etc. When content is described herein as being in real time, thatcontent may be considered live streaming content as described above.Near real time content may represent content that is not necessarilylive streamed, but is presented without being previously recorded. Thecontextual information may be determined based at least in part on oneor more signals indicates the context of the multimedia content, thepresented item(s), and/or the user device viewing the content. Forexample, location tracking information, internet protocol addressinformation, historical customer interaction data, customer preferences,behavioral data, merchant preferences, etc. may be utilized to determinea context and information associated with that context may be utilizedas described herein. With respect to determining where to display theinteractive element 208, the payment processing service provider systemmay determine, utilizing item-recognition data, a relative location ofthe item as depicted in the multimedia content with respect to aviewable window of the user device. For example, the item-recognitiondata may indicate a location of the object as identified in the imagedata, and the interactive element 208 may be generated such that, whendisplayed, the interactive element 208 may be positioned near the objectbut not, in examples, over top of the object. This will allow a user tosee both the object and the interactive element 208, and perceive thatthe object and the interactive element 208 are associated with eachother, while the multimedia content is output. In examples, the firstinteractive element 208 may be placed as an interstitial within the sameapplication in which the item is presented. In another example, theinformation regarding the item may be sent as an electronic message suchas a text message or electronic mail, as an interstitial within anothermobile and/or web application, as a pop-up notification, or on a displaydiscoverable with a gesture (e.g., pinch gesture) or specific keypad,audio, visual, or haptic inputs.

As shown in FIG. 2 , the multimedia content may continue to be outputfrom step 1 or may otherwise progress. At step 2, the multimedia contentmay have progressed to a point where the first item 206 is no longerreferenced (herein not displayed), and instead a second item 210 isreferenced. Utilizing the item recognition techniques as describedelsewhere herein, a time value associated with when the first item 206ceases being referenced and when the second item 208 starts beingreferenced may be determined, and that information may be utilized todetermine when to cease displaying the first interactive element 208 andwhen to start displaying a second interactive element 212 associatedwith the second item. It should also be appreciated that the location ofthe second item 210 with respect to the interactive window has changedfrom the location of the first item 206. This information may beutilized to configure the second interactive element 212 to be displayedin a different location than where the first interactive element 208 wasdisplayed. By so doing, multiple interactive elements may be generatedfor multiple items in the same multimedia content, and each of theinteractive elements may be specifically configured for the itemsassociated with those interactive elements.

As shown in FIG. 2 , the user interface may also include a “shoppingcart” icon 214. The icon 14 may be utilized to provide a visualindication of a number of selected icons and/or other details aboutitems that have been selected for at least potential purchase. The icon214 may be selected to cause a purchasing user interface to bedisplayed. When displayed, item information, as described more fullyherein, may be displayed for the selected items. Additionally, inexamples, purchasing information associated with a user account of theuser may also be displayed. Additionally, the user interface may includea “chat” icon 216, which may be utilized to allow for live chatting witha merchant, such as a merchant associated with the multimedia content.In these examples, the merchant may add an item, such as to a cart,using a physical sensor, e.g., barcode reader. Additionally, the userinterface may include a barcode icon 218, which may be utilized to allowthe user to use a barcode reader, such as a barcode reader associatedwith the user's phone, to scan, for example, coupons or otherinformation associated with the items in question.

It should be understood that while the two items referenced in themultimedia content for the example shown in FIG. 2 are referenced atdifferent times, this disclosure includes the ability to recognize twoor more item referenced concurrently and the generation and display ofmultiple interactive elements at the same time.

FIG. 2 also depicts functionality associated with playback of multimediacontent. That functionality may include selectable elements to controlhow and when the multimedia content is output via a user device. Forexample, the selectable elements may include a play element configuredto cause the multimedia content to be output, a pause element configuredto stop the multimedia content from being output, such as temporarily inexamples, a volume element configured to control the volume of audiooutput and/or to mute audio output, and/or one or more playback speedelements configured to increase or decrease playback speed of themultimedia content and/or to output the multimedia content in reverse.

It should also be understood that while the example of FIG. 2 shows twointeractive elements displayed at different times, this disclosureincludes the generation and/or display of two interactive elementsconcurrently, such as when multiple items are depicted and/or describedat the same time. In these examples, the item identification andlocation processes described herein may be utilized to determine wherethe multiple interactive elements will be located at the same time.

FIG. 3 illustrates an example conceptual diagram showing examples foritem recognition for items referenced in multimedia content. FIG. 3shows a buyer device 102, which may include the same or similarcomponents and perform the same or similar functions as the buyer device102 from FIG. 1 .

At step 1, the buyer device 102 may be presenting multimedia content. Asshown in FIG. 3 , the multimedia content depicts a first item 206 alongwith a person that is speaking. The speech, in this example, is “thisBrand A shirt is a great deal!” An item-recognition component may beutilized to identify the first item 206. For example, theitem-recognition component may analyze the multimedia content and/orrelated data to identify one or more items referenced in the multimediacontent. The item-recognition component may utilize image data of themultimedia content to identify items depicted in the image data. Suchanalysis may include the use of computer vision techniques to identifythe presence of objects in given image data and then to identify theobjects themselves and/or object categories to which the objects belong(e.g., shirt, pants, hat, watch, etc.). Additionally, or alternatively,when the multimedia content includes user speech, speech recognition andnatural language understanding techniques may be utilized to identifythe speech, generate text data representing the speech, and thendetermine an intent and/or purpose of the speech. By doing so, theitem-recognition component 134 may identify items referenced in themultimedia content as well as, in examples, attributes of the items(e.g., color, size, brand, etc.). Additionally, or alternatively, theitem-recognition component 134 may utilize metadata associated with themultimedia content and/or the merchant from which the content wasprovided to identify the items. For example, the merchant device 106 mayprovide metadata indicating the items referenced in the multimediacontent. In examples, one or more other users may have commented on orotherwise provided information in relation to the multimedia content. Inthese and other examples, some or all of this data may be utilized bythe item-recognition component 134 to identify the items and/orattributes associated with the items referenced in the multimediacontent.

With respect to computer vision techniques, computer vision includesmethods for acquiring, processing, analyzing, and understanding imagesand, in general, high-dimensional data from the real world in order toproduce numerical or symbolic information, e.g., in the form ofdecisions. These techniques are described in more detail above withrespect to FIG. 1 . With respect to speech recognition, audio datacorresponding to the user speech may be processed by an automatic speechrecognition component, which may compare attributes of the audio data toreference attributes to determine that given sounds correspond to givenwords. In this way, the automatic speech recognition component maygenerate text data that represents the words of the user speech. Anatural language understanding component may utilize the text data todetermine keywords or other elements of the text data that correspond toa purpose for the speech. In the example in FIG. 3 , this process mayresult in the system determining that the user speech of “this Brand Ashirt is a great deal!” indicates that there is an item being displayedand that item is a shirt, with one of the shirt attributes being “BrandA.” This information may be utilized to recognize the item and/orattributes associated with the item. In addition to the image analysisdescribed above, computer vision techniques may be utilized to identifygestures of people depicted in multimedia content. These gestures mayindicate that the person is describing a given item and/or identifyingthat item as being available for purchase. This information may beutilized to recognize items and/or to determine which items areavailable for purchase.

At step 2, the item-recognition data may be utilized by aninteractive-element generator to generate and place an interactiveelement 208 associated with the first item 206. The interactive element208 may include or otherwise reference some or all of theitem-recognition data, such as the “Brand A” attribute, the item type“shirt,” as well as other information such as current inventory count.Interactive elements may include any selectable element that may bepresented to a user and be configured to receive user input indicatingselection. E-commerce tags can be a type of interactive element and maybe selectable portions associated with multimedia content that mayinclude e-commerce specific actions, such as purchasing actions, iteminformation actions, and/or other shopping related actions, for example.

FIG. 4A illustrates an example user device 102 displaying a firstexample interactive element overlaid, or otherwise embedded, onmultimedia content while the users are viewing the multimedia content.Interactive elements may include any selectable element that may bepresented to a user and be configured to receive user input indicatingselection. E-commerce tags can be a type of interactive element and maybe selectable portions associated with multimedia content that mayinclude e-commerce specific actions, such as purchasing actions, iteminformation actions, and/or other shopping related actions, for example.In examples, the interactive element may be placed as an interstitialwithin the same application in which the item is presented. In anotherexample, the information regarding the item may be sent as an electronicmessage such as a text message or electronic mail, as an interstitialwithin another mobile and/or web application, as a pop-up notification,or on a display discoverable with a gesture (e.g., pinch gesture) orspecific keypad, audio, visual, or haptic inputs. The example providedin FIG. 4A shows a first interactive element 402 overlaid on multimediacontent referencing a shirt. The first interactive element 402 may be aselectable element, such as a link, that may cause one or more actionsto be performed when selected. In some examples, the first interactiveelement 402 may be considered selected when touch input is received atthe user device 102 on a portion of the display of the user device 102corresponding to where the first interactive element 402 is displayed.Other forms of user input such as clicking or keyboard navigation to thefirst interactive element 402 may also be received. In one embodiment,after the action is performed, the multimedia content may automaticallyand immediately be replaced with a summary image displaying thecorresponding summary information on initiation of the swiping ordragging motion, such as the cart, with an optional link for the user toreturn to the multimedia content after reviewing the summary orotherwise performing other actions. In another embodiment, the cart maybe shown as a picture in picture mode while the user navigates themultimedia content, or as a separate window alongside the multimediacontent with the multimedia content adjusted to accommodate the separatewindow showing cart information. In yet another embodiment, themultimedia content is not replaced with another window and the summaryinformation may be tracked as a background process and in a separatedata structure with user and merchant identity and displayed after themultimedia content is over, for example. In one example, the cartinformation can also be sent as an electronic message, such as email ortext, or within a mobile application of the user or merchant. Themerchant may be interested in seeing such information to offerincentives to the customer and encourage the user to finalize thepurchase.

The first interactive element 402 having the input type described inFIG. 4A may be selected and utilized for presentation of interactiveelements based at least in part on a device type of the user device 102,user preference data, and/or as requested by a merchant and/or aprovider of the multimedia content. For example, when the user device102 has user input capabilities such as a touchscreen, a mouse, and/or akeyword, the first interactive element 402 may be selected for displayon the user device 102. By way of additional example, user preferencedata may indicate that the user typically provides certain user inputtypes that indicate the first interactive element 402 would be favorablefor use as opposed to one or more other interactive element types.

FIG. 4B illustrates an example user device 102 displaying a secondexample interactive element overlaid on multimedia content. The exampleprovided in FIG. 4B shows a second interactive element 404 overlaid onmultimedia content referencing a shirt. The second interactive element404 may be a readable element, such as a quick response code or abarcode, that may cause one or more actions to be performed when scannedand/or when another device indicates that it has scanned the quickreference code. Interactive elements may include any selectable elementthat may be presented to a user and be configured to receive user inputindicating selection. E-commerce tags can be a type of interactiveelement and may be selectable portions associated with multimediacontent that may include e-commerce specific actions, such as purchasingactions, item information actions, and/or other shopping relatedactions, for example. When generating such interactive elements 404, theactions can be defined and customized based on the device or useridentity scanning the element. In some examples, the second interactiveelement 404 may be considered selected when the user device 102 and/oranother device indicates that the quick response code has been scanned.For example, the quick response code may be displayed on a first devicethat is also displaying the multimedia content. A second device may scanthe quick response code and the indication that the quick response codehas been scanned may be received from the second device, in examples. Itshould be understood that while quick response codes are utilized hereas an example, and other type of scannable, readable, and/oridentifiable element may be utilized. In one embodiment, after theaction is performed, the multimedia content may automatically andimmediately be replaced with a summary image displaying thecorresponding summary information on initiation of the swiping ordragging motion, such as the cart, with an optional link for the user toreturn to the multimedia content after reviewing the summary orotherwise performing other actions. In another embodiment, the cart maybe shown as a picture in picture mode while the user navigates themultimedia content, or as a separate window alongside the multimediacontent with the multimedia content adjusted to accommodate the separatewindow showing cart information. In yet another embodiment, themultimedia content is not replaced with another window and the summaryinformation may be tracked as a background process and in a separatedata structure with user and merchant identity and displayed after themultimedia content is over, for example. In one example, the cartinformation can also be sent as an electronic message, such as email ortext, or within a mobile application of the user or merchant. Themerchant may be interested in seeing such information to offerincentives to the customer and encourage the user to finalize thepurchase.

The second interactive element 404 having the input type described inFIG. 4B may be selected and utilized for presentation of interactiveelements based at least in part on a device type of the user device 102,user preference data, and/or as requested by a merchant and/or aprovider of the multimedia content. For example, when the user device102 lacks certain capabilities such as a touchscreen, a mouse, and/or akeyword, the second interactive element 404 may be selected for displayon the user device 102. To this end, in one implementation, the secondinteractive element 404 is generated based on device characteristics ofthe device on which the content is displayed and/or another device usedfor reading the interactive element and thereby determining whetherfactors, such as form factor, operating system, reading capabilities,user preferences, etc., and accordingly configuring the secondinteractive element for that device. By way of additional example, userpreference data may indicate that the user typically provides certainuser input types that indicate the second interactive element 404 wouldbe favorable for use as opposed to one or more other interactive elementtypes.

FIG. 4C illustrates an example user device 102 displaying a thirdexample interactive element overlaid on multimedia content. The exampleprovided in FIG. 4C shows a third interactive element 406 overlaid onmultimedia content referencing a shirt. The third interactive element406 may be an indicator that voice input may be received to cause one ormore actions to be performed. Interactive elements may include anyselectable element that may be presented to a user and be configured toreceive user input indicating selection. E-commerce tags can be a typeof interactive element and may be selectable portions associated withmultimedia content that may include e-commerce specific actions, such aspurchasing actions, item information actions, and/or other shoppingrelated actions, for example. In one embodiment, after the action isperformed, the multimedia content may automatically and immediately bereplaced with a summary image displaying the corresponding summaryinformation on initiation of the swiping or dragging motion, such as thecart, with an optional link for the user to return to the multimediacontent after reviewing the summary or otherwise performing otheractions. In another embodiment, the cart may be shown as a picture inpicture mode while the user navigates the multimedia content, or as aseparate window alongside the multimedia content with the multimediacontent adjusted to accommodate the separate window showing cartinformation. In yet another embodiment, the multimedia content is notreplaced with another window and the summary information may be trackedas a background process and in a separate data structure with user andmerchant identity and displayed after the multimedia content is over,for example. In one example, the cart information can also be sent as anelectronic message, such as email or text, or within a mobileapplication of the user or merchant. The merchant may be interested inseeing such information to offer incentives to the customer andencourage the user to finalize the purchase.

In some examples, the third interactive element 406 may be consideredselected when the user device 102 and/or a remote system such as aspeech-processing system indicates that voice input indicates an intentto select the third interactive element 406. The audio input may includespecific trigger words created specifically to enable an ecommerceexperience between the merchant and the customers. For example, theaudio input may only be received when the interactive element 406 isdisplayed on the device. The audio then is recorded from that time stampto a future timestamp (e.g., end of the video or end of the indicatoridentifying the interactive element 406). Such time stamps may definerespective time offsets from the start of an audio recording or fromanother particular position in the audio recording. During therecording, it is identified whether the customer provides an intent topurchase, e.g., through use of trigger words, such as “Add this item tomy cart,” “buy in green size small” and accordingly maps those triggerwords to the specific interactive element, adds those specific items toan online shopping cart. The trigger words can also be predefined by themerchant or customer and stored in respective profiles, and algorithmthat analyzes the audio recording to create an online shopping cart. Inanother example, the trigger words can also be counted or furtherfiltered to determine additional specifics, such as how many items doesthe customer want, what color, what size, etc.

In some implementations, the methods and systems described herein can beintegrated with voice services (e.g. Amazon's ALEXA®, Apple's SIRI®, orMicrosoft's CORTANA®) through specific API calls to such services fromwithin the multimedia interface. The present methods and systems canintegrate with the “wake words” for invoking their respective voiceservice, ecommerce and fulfillment channels. For example, speakerrecognition techniques may be utilized to determine user profilesassociated with users that provide user utterances to user devices forperforming one or more of the operations described herein. Thedetermined user profile may be utilized to customize the interactiveelements described herein, functionality that occurs when an interactiveelement is selected, and/or the information that is displayed in thepurchasing user interface. The voice interface may also be utilized toprovide the user inputs described herein. In addition, the voiceinterface may be associated with an online platform, and that platformmay be utilized to fulfill item purchasing.

The third interactive element 406 having the input type described inFIG. 4C may be selected and utilized for presentation of interactiveelements based at least in part on a device type of the user device 102,user preference data, and/or as requested by a merchant and/or aprovider of the multimedia content. For example, when the user device102 lacks certain capabilities such as a touchscreen, a mouse, and/or akeyword, the third interactive element 406 may be selected for displayon the user device 102. By way of additional example, user preferencedata may indicate that the user typically provides certain user inputtypes that indicate the third interactive element 406 would be favorablefor use as opposed to one or more other interactive element types.

FIG. 4D illustrates an example user device 102 displaying a fourthexample interactive element overlaid on multimedia content. The exampleprovided in FIG. 4D shows a fourth interactive element 408 overlaid onmultimedia content referencing a shirt. The fourth interactive element408 may be an indicator that gesture input, when received, may cause oneor more embedded actions to be performed. Interactive elements mayinclude any selectable element that may be presented to a user and beconfigured to receive user input indicating selection. E-commerce tagscan be a type of interactive element and may be selectable portionsassociated with multimedia content that may include e-commerce specificactions, such as purchasing actions, item information actions, and/orother shopping related actions, for example. The gesture may be aswiping or pinching gesture swiping gesture in which the interactiveelement is grabbed (e.g., by a finger press on a touchscreen or a cursorclick and hold via mouse input) and dragged, swiped, clicked or tappedin a specific direction, to allow the embedded action (for example additem to a cart) to be initialized. In one embodiment, after the actionis performed, the multimedia content may automatically and immediatelybe replaced with a summary image displaying the corresponding summaryinformation on initiation of the swiping or dragging motion, such as thecart, with an optional link for the user to return to the multimediacontent after reviewing the summary or otherwise performing otheractions. In another embodiment, the cart may be shown as a picture inpicture mode while the user navigates the multimedia content, or as aseparate window alongside the multimedia content with the multimediacontent adjusted to accommodate the separate window showing cartinformation. In yet another embodiment, the multimedia content is notreplaced with another window and the summary information may be trackedas a background process and in a separate data structure with user andmerchant identity and displayed after the multimedia content is over,for example. In one example, the cart information can also be sent as anelectronic message, such as email or text, or within a mobileapplication of the user or merchant. The merchant may be interested inseeing such information to offer incentives to the customer andencourage the user to finalize the purchase.

In some examples, the fourth interactive element 408 may be consideredselected when the user device 102 indicates that image data collected,for example, by a camera of the user device 102, depicts movement of auser in a pattern that corresponds to a reference pattern associatedwith selection of the fourth interactive element 408.

The fourth interactive element 408 having the input type described inFIG. 4D may be selected and utilized for presentation of interactiveelements based at least in part on a device type of the user device 102,user preference data, and/or as requested by a merchant and/or aprovider of the multimedia content. For example, when the user device102 lacks certain capabilities such as a touchscreen, a mouse, and/or akeyword, the fourth interactive element 408 may be selected for displayon the user device 102. By way of additional example, user preferencedata may indicate that the user typically provides certain user inputtypes that indicate the fourth interactive element 408 would befavorable for use as opposed to one or more other interactive elementtypes.

As can be seen in FIGS. 4A-4D, a user may provide user input indicatingselection of the interactive elements 402-408. Regardless of the type ofinteractive element utilized, when user input data indicates selectionof such interactive elements, a process may include displaying apurchasing user interface based at least in part on the received userinput data indication selection of the interactive element.Additionally, the interactive elements may be adjusted for visibility,positioning, etc. and/or may be customized for content and interpretedwhen selected based on customer authentication. The modified view ofinteractive elements can be based on what we know about the customer,such as from historical account data, from cookie sessions of theconsumer and the device they're using, etc. In this way the presentationof interactive elements may be customizable for the user in question.

FIGS. 5-12 illustrate processes for e-commerce tags in multimediacontent and customized e-commerce tags in real-time multimedia content.The processes described herein are illustrated as collections of blocksin logical flow diagrams, which represent a sequence of operations, someor all of which may be implemented in hardware, software or acombination thereof. In the context of software, the blocks mayrepresent computer-executable instructions stored on one or morecomputer-readable media that, when executed by one or more processors,program the processors to perform the recited operations. Generally,computer-executable instructions include routines, programs, objects,components, data structures and the like that perform particularfunctions or implement particular data types. The order in which theblocks are described should not be construed as a limitation, unlessspecifically noted. Any number of the described blocks may be combinedin any order and/or in parallel to implement the process, or alternativeprocesses, and not all of the blocks need be executed. For discussionpurposes, the processes are described with reference to theenvironments, architectures and systems described in the examplesherein, such as, for example those described with respect to FIGS. 1-4D,13, and 14 , although the processes may be implemented in a wide varietyof other environments, architectures and systems.

FIG. 5 illustrates an example process 500 for determining and utilizinginventory data in real time in association with multimedia contentrepresenting an item. The order in which the operations or steps aredescribed is not intended to be construed as a limitation, and anynumber of the described operations may be combined in any order and/orin parallel to implement process 500.

At block 502, the process 500 may include identifying an item associatedwith multimedia content. For example, an item-recognition component mayanalyze the multimedia content and/or related data to identify one ormore items referenced in the multimedia content. For example, theitem-recognition component may utilize image data of the multimediacontent to identify items depicted in the image data. Such analysis mayinclude the use of computer vision techniques to identify the presenceof objects in given image data and then to identify the objectsthemselves and/or object categories to which the objects belong (e.g.,shirt, pants, hat, watch, etc.). Additionally, or alternatively, whenthe multimedia content includes user speech, speech recognition andnatural language understanding techniques may be utilized to identifythe speech, generate text data representing the speech, and thendetermine an intent and/or purpose of the speech. By doing so, theitem-recognition component may identify items referenced in themultimedia content as well as, in examples, attributes of the items(e.g., color, size, brand, etc.). Additionally, or alternatively, theitem-recognition component may utilize metadata associated with themultimedia content and/or the merchant from which the content wasprovided to identify the items. For example, a merchant device mayprovide metadata indicating the items referenced in the multimediacontent. In examples, one or more other users may have commented on orotherwise provided information in relation to the multimedia content. Inthese and other examples, some or all of this data may be utilized bythe item-recognition component to identify the items and/or attributesassociated with the items referenced in the multimedia content.

At block 504, the process 500 may include identifying a merchantassociated with the multimedia content. For example, the multimediacontent may include metadata and/or otherwise an indication of a sourceof the multimedia content, and that source may be associated with themerchant. In one implementation, the service provider provides theplatform for one or more of: processing payments for one or moremerchants, including this merchant, maintaining inventory, providingpayroll services, providing lending services, etc.

At block 506, the process 500 may include receiving inventory dataassociated with the merchant. For example, a payment processing serviceprovider may store inventory data associated with the merchant and/or asystem associated with the merchant may provide the inventory data inreal time or near real time to the payment processing service provider.In another example, the payment processing service provider may, throughAPI calls or through web-scraping, access third party inventorydatabases to obtain the most current inventory status.

At block 508, the process 500 may include determining whether theinventory data indicates the item is available from the merchant. Forexample, an item-information component may query the merchant system forinventory data indicating a current inventory of the item(s), such as ata time that the multimedia content is being output. In examples, themerchant system may return the inventory data and that inventory datamay be utilized to inform a customer of a current inventory of items asavailable from the merchant. In other examples, such as where theinventory data indicates the item is out of stock and/or when userpreferences indicate that a different merchant is preferred by thecustomer, an indication of the current inventory of one or more othermerchants may be retrieved and displayed on the user device. In someexamples, items comparable to the items being references may also besearched for availability at this merchant or comparable merchants,and/or, and appropriate substitutions at the same merchant or adifferent merchant may be recommended accordingly

In instances where the inventory data indicates the item is availablefrom the merchant, the process 500 may include, at block 510, generatingan interactive element that includes an indicator of an inventory countof the item. For example, the interactive element may be generated asdescribed herein and may include information associated with the item,such as identifying information about the item. In addition, anindicator of the current inventory count may also be displayed. Inexamples, the inventory count may be “live” or may otherwise change ascustomers purchase an instance of the item from the merchant. In thisway, customer may perceive how quickly the item is selling. Thegeneration and/or placement of interactive elements is described in moredetail with respect to FIGS. 2 and 3 , above. Interactive elements mayinclude any selectable element that may be presented to a user and beconfigured to receive user input indicating selection. E-commerce tagscan be a type of interactive element and may be selectable portionsassociated with multimedia content that may include e-commerce specificactions, such as purchasing actions, item information actions, and/orother shopping related actions, for example.

At block 512, the process 500 may include causing a user device on whichthe multimedia content is output to display the interactive element. Theinteractive element may be displayed, such as by way of an overlay, andmay be displayed while the item is being referenced by the multimediacontent, when output.

Returning to block 508, in instances where the inventory data indicatesthe item is not available from the merchant, the process 500 mayinclude, at block 514, identifying one or more other merchants fromwhich the item is currently available. For example, the paymentprocessing service provider may have access to inventory data from oneor more other merchants. An identifier of the item may be utilized toquery inventory data from the one or more other merchants, and thatinformation may be utilized by the payment processing service providerto determine which of the other merchants have the item currently instock.

In other examples, the link associated with the interactive element maybe broken or otherwise not available. In these examples, the iteminformation may be utilized to find substitute items and/or merchantsand links to purchasing user interfaces associated with those substitutemerchants may be provided.

At block 516, the process 500 may include generating the interactiveelement that includes an indicator of the one or more other merchants.For example, the interactive element may provide an indication that theitem is not currently in stock from the merchant associated with themultimedia content and may direct the customer to select the interactiveelement to see other merchants that do have the item in stock. In otherexamples, the interactive element may include an identifier of at leastone of the other merchants that have the item in stock.

At block 518, the process 500 may include causing the user device onwhich the multimedia content is output to display the interactiveelement. For example, the interactive element may be displayed, such asby way of an overlay, and may be displayed while the item is beingreferenced by the multimedia content, when output.

FIG. 6 illustrates an example process 600 for determining whether toaggregate item selections during presentation of multimedia content. Theorder in which the operations or steps are described is not intended tobe construed as a limitation, and any number of the described operationsmay be combined in any order and/or in parallel to implement process600.

At block 602, the process 600 may include causing output of multimediacontent with an interactive element associated with an item such thatthe interactive element is overlaid on the multimedia content. Forexample, the multimedia content may be output and may include imagesand/or audio that reference one or more items. While the multimediacontent is output, or during at least a portion of the output multimediacontent, the interactive element may also be displayed, for example asan overlay to the content, such that a user may see the multimediacontent and the interactive element. Interactive elements may includeany selectable element that may be presented to a user and be configuredto receive user input indicating selection. E-commerce tags can be atype of interactive element and may be selectable portions associatedwith multimedia content that may include e-commerce specific actions,such as purchasing actions, item information actions, and/or othershopping related actions, for example.

At block 604, the process 600 may include determining whether one ormore user-configurable options indicate a preference for aggregatingitem selections. For example, user-provided preferences may be utilizedto determine whether a user desires to wait until the cessation of themultimedia content to have a purchasing user interface displayed.

In instances where the user-configurable options do not indicate thepreference for aggregating item selections, the process 600 may include,at block 606, determining whether user history indicates a preferencefor aggregating selections. For example, historical transaction data maybe utilized to determine if the user typically purchases more than oneitem referenced in the same multimedia content.

In instances where the user history indicates the preference foraggregating selections and/or where the user-configurable optionsindicate the preference for aggregating item selections, the process 600may include, at block 608, storing data representing engagement withinteractive elements during output of the multimedia content. Forexample, the system may determine to refrain from causing a purchasinguser interface to be displayed in response to selection of aninteractive element. Instead, the system may store data representing theselection(s) of interactive element(s) while the multimedia content isoutput. In still other examples, the purchasing interface may bedisplayed but may not interfere with the output of the multimediacontent. In some examples, multiple multimedia content may be displayed,even in instances where the multimedia content is associated withdifferent merchants and/or content providers, such as multiple videos,before the purchasing user interface is displayed.

At block 610, the process 600 may include refraining from displaying apurchasing user interface until occurrence of an event, such ascessation of the multimedia content, a request is received, a spendinglimit is exceeded, a designated time, etc. For example, instead ofstopping the multimedia content from being output and/or causing anotherwindow to open that hinders the customer's ability to view the remainderof the multimedia content, the system may refrain from doing so, and mayinstead, in examples, cause an indictor to be presented while themultimedia content is displayed that indicates the item has beenselected and/or a number of selected items.

At block 612, the process 600 may include displaying the purchasing userinterface with aggregated item selections in response to cessation ofthe multimedia content. In this example, identifying informationassociated with each of the selected items may be displayed and/or maybe available for display such that a customer may purchase all of theselected items at the same time.

Returning to block 606, in instances where the user history does notindicate the preference for aggregating selections, the process 600 mayinclude, at block 614, displaying the purchasing user interface with theselected item. In this example, upon selection of the interactiveelement, the purchasing user interface may be displayed with identifyinginformation for the item that was selected. The multimedia content maybe caused to stop being output, or, in examples, the focus on the userdevice may be on the purchasing user interface instead of the multimediacontent. Even though the user interface is described to be a purchaseuser interface with functionalities to allow payment for the selecteditems, it will be understood that the user interface can be configuredto offer other views, such as summary information regarding selecteditems, additional information links regarding the selected items,separate purchase links for each of the selected items, etc.

FIG. 7 illustrates an example process 700 for modifying interactiveelements based at least in part on user preference data. The order inwhich the operations or steps are described is not intended to beconstrued as a limitation, and any number of the described operationsmay be combined in any order and/or in parallel to implement process700.

At block 702, the process 700 may include receiving text datacorresponding to one or more comments to multimedia content. Forexample, a feedback component of the payment processing service providersystem may be configured to receive feedback data associated with theinteractive elements. In at least one example, one or more users canrespond to content and provide feedback, for example, via comments(which can include text, images, emojis, etc.), interactions with abutton or other actuation mechanism (e.g., like, dislike, funny, love,etc.), and so on, for example on a content provider's media platform(e.g., a social networking platform, a microblog, a blog, video sharingplatform, a music sharing platform, etc.) that enables user interactionand engagement through comments, posts, messages on electronic bulletinboards, messages on a social networking platform, and/or any other typesof messages. The content provider can enable users of the media platformto interact with one another, (e.g., through creating messages, postingcomments, etc.). In some embodiments, the content media platform mayalso refer to an application or webpage of an e-commerce or retailorganization that offers products and/or services. Such websites canprovide an online “form” to complete before or after the products orservices are added to a virtual cart. The online form may include one ormore fields to receive user interaction and engagement, such asquestions regarding the order, feedback on previous orders. Suchresponses can be posted in near real-time. The feedback data mayindicate positive and/or negative comments about the interactiveelements, display of the interactive elements, and/or functionalityassociated with selection of the interactive elements. The feedback datamay include the text data corresponding to such feedback.

At block 704, the process 700 may include analyzing the text data forkeywords associated with the item. For example, natural languageunderstanding techniques may be utilized to parse the text data andidentify words that may be important to the context of the comment(s)and the multimedia content. By way of example, the comment “this shirtalso comes in green” may be processed and annotated with tags todetermine the semantic interpretation of the comment and that thecomment is about an attribute of the shirt, here color, and the specificattribute is “green.” Further additional identifiers can be used todetermine identity of “this shirt.”

At block 706, the process 700 may include identifying the item and/orattributes associated with the item from the keywords. For example,keywords that are associated with reference attributes of items may beidentified and utilized to determine one or more of the attributes ofthe items. Those attributes may include any physical detail about theitems and/or one or more details associated with the items.

At block 708, the process 700 may include determining whether aninteractive element has been generated for one or more items representedin the multimedia content. For example, the multimedia content may havealready been analyzed by the payment processing service provider system,which may have already generated the interactive element. In otherexamples, the multimedia content may not yet have been analyzed and/oran interactive element is not associated with the multimedia content.

In instances where the interactive element has not been generated, theprocess 700 may include, at block 710, generating the interactiveelement indicating the identified item and/or the item attributes. Forexample, an interactive-element generator of the payment processingservice provider system may be configured to utilize the data receivedand/or determined by the item-recognition component and/or theitem-information component to generate data representing an interactiveelement. The interactive element may be configured such that, when themultimedia content is output on the user device, the interactive elementis also presented, such as in the form of an overlay. The interactiveelement may be, in examples, specific to the multimedia content, theitems referenced therein, the item attributes, and/or user preferences.For example, utilizing the data received and/or determined as describedherein, the interactive-element generator may determine a type ofinteractive element to generate. Interactive element types may include,for example, a selectable link, a quick response code (“QR code”), anindicator that voice input for selection of the interactive element isenabled, an indicator that gesture input for selection of theinteractive element is enabled, etc. It should be understood that whileseveral examples of element types have been provided herein, thisdisclosure includes any element type that allows user input to bereceived. Determining the type of interactive element to associate withgiven multimedia content may be based at least in part on a device typeof the user device. For example, if the device type indicates that thedevice includes a camera, gesture-based interactive elements may beutilized, or if the device type indicates the device does not include atouchscreen, the interactive element may be configured to accept userinput other than touchscreen input. Additionally, or alternatively,purchase history associated with a user profile that is being utilizedto view the multimedia content may be utilized to determine past userinput types, and that information may be utilized to determine the typeof interactive element to be generated. Interactive elements may includeany selectable element that may be presented to a user and be configuredto receive user input indicating selection. E-commerce tags can be atype of interactive element and may be selectable portions associatedwith multimedia content that may include e-commerce specific actions,such as purchasing actions, item information actions, and/or othershopping related actions, for example.

In addition to the type of interactive element, the interactive-elementgenerator may be configured to determine one or more other aspectsassociated with the interactive element, such as when to display theinteractive element with respect to the multimedia content, where todisplay the interactive element with respect to a viewable window of auser device, an amount and/or type of item detail to display, and/orfunctionality that occurs when the interactive element is selected, forexample. By way of example, the interactive-element generator maydetermine when to display the interactive element based at least in parton data indicating when the item begins being referenced in themultimedia content and when the item ceases being referenced. Forexample, given content may be two minutes in length, but the item maynot start being referenced until the 30-second mark and then ceasesbeing referenced at the 1-minute mark. Utilizing the item-recognitiondata described herein, the interactive-element generator may generatethe interactive element to be configured to be displayed only during thetime frame when the item is referenced. With respect to determiningwhere to display the interactive element, the interactive-elementgenerator may determine, utilizing the item-recognition data, a relativelocation of the item as depicted in the multimedia content with respectto a viewable window of the user device. For example, theitem-recognition data may indicate a location of the object asidentified in the image data, and the interactive element may begenerated such that, when displayed, the interactive element may bepositioned near the object but not, in examples, over top of the object.This will allow a user to see both the object and the interactiveelement, and perceive that the object and the interactive element areassociated with each other, while the multimedia content is output.

At block 712, the process 700 may include causing display of theinteractive element while at least a portion of the multimedia content soutput. For example, the interactive element may be displayed, such asby way of an overlay, and may be displayed while the item is beingreferenced by the multimedia content, when output.

Returning to block 708, in instances where the interactive element hasnot been generated, the process 700 may include, at block 714, modifyingthe interactive element to include the identified item and/or the itemattributes. For example, one or more portions of the interactive elementmay be dynamically changed and/or updated to include the identifier itemand/or the item attributes. By way of example, an original interactiveelement may have included that the item was “black shirt,” but with theinformation gathered and/or determined as described above, theinteractive element may be changed to “black Brand A t-shirt.” It shouldbe understood that when this disclosure discusses an interactive elementchanging and/or being modified, such disclosure includes generating datarepresenting an interactive element with the changed information. Insome implementations, the modification may be triggered by inventory oravailability status of the item, for example, if the t-shirt is notavailable in black, the color is updated to alternate colors. In anotherimplementation, the modification may be triggered by customer ormerchant-based rules (either explicitly specified by such parties orimplicitly determined based on historical transactions) to modify itemdescriptions. For example, customer purchase history may be used todynamically modify the item description to the size that the customer(viewing the content) usually buys. Alternatively, the merchantpreferences may be used to modify the item description to the color themerchant wants to sell first. In another implementation, themodification can be initiated based on the contextual rules, where thecontext is derived from factors, such as location of the merchant,location of the customer, time of the day, what customers are currentlybuying, what merchants are currently selling, items with incentives suchas coupons, and so on.

As described herein, the generation and/or modification of newinteractive elements, which may be generated on-the-fly in someembodiments, that are used to display selectable links, quick referencecodes, tags, etc. while multimedia content is being output on a screenof a user device represents a computer-centric solution to thecomputer-centric problem of online platform utilization. Further, thetechniques described herein include the generation of new data, such asthe link data described herein which is specific to given multimediacontent and is configured to be displayed, in examples, differently fordifferent user devices. By doing so, online platform marketplaces may beaugmented such that information displayed to potential customers is moreaccurate, tailored to the specific customer, is presented in a timesensitive manner, and provides functionality that could only exist in acomputer-centric environment. Commands to display selectable elements asdescribed herein may also be configured to cause applications on userdevices to initiate and/or activate and cause display of time-sensitiveinformation, such as without user input. By so doing, a material changein the user devices themselves is achieved such that the user devicesmay perform functions that otherwise would not have been possible absentthe computer-centric solutions described herein.

At block 716, the process 700 may include causing display of themodified interactive element. For example, the modified interactiveelement may be displayed, such as by way of an overlay, and may bedisplayed while the item is being referenced by the multimedia content,when output.

FIG. 8 illustrates an example process 800 for utilizing user preferencesfor modifying display of item information and for pre-populatingpurchasing user interfaces. The order in which the operations or stepsare described is not intended to be construed as a limitation, and anynumber of the described operations may be combined in any order and/orin parallel to implement process 800.

At block 802, the process 800 may include receiving multimedia content.For example, a content component of the payment processing serviceprovider system may be configured to receive the multimedia contentand/or to retrieve the multimedia content. For example, a merchantsystem or other system may push the multimedia content to the paymentprocessing service provider system without a specific request for suchcontent. In other examples, the content component may query one or moreother systems for the multimedia content. In still other examples, thecontent component may receive an indication that multimedia contentassociated with a given merchant has been requested to be output on auser device associated with a customer. In these examples, the contentcomponent may query for an instance of the multimedia content andperform the techniques for generating an interactive element overlay,such as prior to the multimedia content being output on the user device.

At block 804, the process 800 may include identifying one or more itemsrepresented in the multimedia content. For example, an item-recognitioncomponent of the payment processing service provider system may analyzethe multimedia content and/or related data to identify one or more itemsreferenced in the multimedia content. For example, the item-recognitioncomponent may utilize image data of the multimedia content to identifyitems depicted in the image data. Such analysis may include the use ofcomputer vision techniques to identify the presence of objects in givenimage data and then to identify the objects themselves and/or objectcategories to which the objects belong (e.g., shirt, pants, hat, watch,etc.). Additional details on the use of computer vision techniques isprovided below. Additionally, or alternatively, when the multimediacontent includes user speech, speech recognition and natural languageunderstanding techniques may be utilized to identify the speech,generate text data representing the speech, and then determine an intentand/or purpose of the speech. By doing so, the item-recognitioncomponent may identify items referenced in the multimedia content aswell as, in examples, attributes of the items (e.g., color, size, brand,etc.). Additionally, or alternatively, the item-recognition componentmay utilize metadata associated with the multimedia content and/or themerchant from which the content was provided to identify the items. Forexample, the merchant system may provide metadata indicating the itemsreferenced in the multimedia content. In examples, one or more otherusers may have commented on or otherwise provided information inrelation to the multimedia content. In these and other examples, some orall of this data may be utilized by the item-recognition component toidentify the items and/or attributes associated with the itemsreferenced in the multimedia content.

At block 806, the process 800 may include determining whether userpreferences have been determined for a user profile being utilized toview the multimedia content. For example, user input and/or pasttransactions associated with a user profile may indicate one or moreuser preferences.

In instances where user preferences have been determined, the process800 may include, at block 808, determining item information to displayutilizing the user preferences. With respect to determining an amountand/or type of item detail to display, an interactive-element generatormay utilize the item information from the item-information component todetermine attributes associated with the referenced item. In someexamples, all of the attributes may be included in the interactiveelement. However, in other examples, only a portion of the attributesmay be included. For example, utilizing the historical data associatedwith the user profile, one or more user preferences may be receivedand/or determined, and those user preferences may inform the selectionof which item information to include in the interactive element. Forexample, the historical data may indicate that the user associated withthe user profile at issue purchases more items with a certain degree ofitem detail and/or certain types of item detail provided. By way ofadditional example, the historical data may be data associated with morethan (or other than) the user profile at issue, such as historical dataassociated with customers of the merchant, customers of differentmerchants, and/or customers in general.

At block 810, the process 800 may include determining an interactiveelement type to display utilizing the user preferences. For example,utilizing the data received and/or determined as described herein, theinteractive-element generator may determine a type of interactiveelement to generate. Interactive element types may include, for example,a selectable link, a QR code, an indicator that voice input forselection of the interactive element is enabled, an indicator thatgesture input for selection of the interactive element is enabled, etc.It should be understood that while several examples of element typeshave been provided herein, this disclosure includes any element typethat allows user input to be received. Determining the type ofinteractive element to associate with given multimedia content may bebased at least in part on a device type of the user device. For example,if the device type indicates that the device includes a camera,gesture-based interactive elements may be utilized, or if the devicetype indicates the device does not include a touchscreen, theinteractive element may be configured to accept user input other thantouchscreen input. Additionally, or alternatively, purchase historyassociated with a user profile that is being utilized to view themultimedia content may be utilized to determine past user input types,preferred payment options (such as previously recorded payment card onfile) and that information may be utilized to determine the type ofinteractive element to be generated. Interactive elements may includeany selectable element that may be presented to a user and be configuredto receive user input indicating selection. E-commerce tags can be atype of interactive element and may be selectable portions associatedwith multimedia content that may include e-commerce specific actions,such as purchasing actions, item information actions, and/or othershopping related actions, for example.

Returning to block 806, in instances where user preferences have notbeen determined, the process 800 may include, at block 812, determiningitem information to display utilizing default preferences. For example,a set of default preferences may be utilized to provide a given amountand/or type of item detail.

At block 814, the process 800 may include determining the interactiveelement type to display utilizing device capability data. This may beperformed in the same or a similar manner to the operations described atblock 810.

At block 816, the process 800 may include causing the interactiveelement to be displayed with the multimedia content. For example, theinteractive element may be displayed, such as by way of an overlay, andmay be displayed while the item is being referenced by the multimediacontent, when output.

At block 818, the process 800 may include receiving user input dataindicating selection of the interactive element. For example, a user mayprovide input to the user device, which may generate user input dataindicating selection of the interactive element.

At block 820, the process 800 may include causing display of apurchasing user interface with prepopulated item information and/orpurchasing information. For example, one or more user-input fields ofthe purchasing user interface may be prepopulated based at least in parton some or all of the data discussed herein. For example, data from theuser profile may be utilized to prepopulate attributes and/or optionsassociated with a selected item. Additionally, or alternatively, theitem information determined from the multimedia content may be utilizedto prepopulate the item attributes. Additionally, payment informationfrom past transactions associated with the user profile may be utilizedto prepopulate payment options, delivery addresses, etc. on thepurchasing user interface. In this way, upon selection of theinteractive element(s), the purchasing user interface may beautomatically displayed and prepopulated with item and paymentinstrument information such that a user may need to only confirm thepurchase without providing any additional input to acquire the item. Insome examples, the payment processing service provider system may beassociated with the purchasing user interface, and in these examples,the system may allow for items the user has expressed interest in to bestored in association with one or more applications associated with thesystem. The application may allow for synchronization across multiplemerchants and may allow for finalization of purchasing, even acrossmultiple merchants, at a later time. The application may also providevarious payment fulfillment options, which may be based at least in parton the item being purchased, the merchant(s) in question, userpreferences, etc. In some examples, fulfillment options may include alending option, where the payment processing service provider systemprovides an option to provide a loan, which may allow the user topurchase the item on conditions of the loan. The option to providelending options may be based at least in part on historical dataassociated with the customer, the merchant, and/or based on coupons orother identifying information indicating that lending is available for agiven transaction.

FIG. 9 illustrates an example process 900 for electronic commerce(“e-commerce”) tags in multimedia content. The order in which theoperations or steps are described is not intended to be construed as alimitation, and any number of the described operations may be combinedin any order and/or in parallel to implement process 900.

At block 902, the process 900 may include receiving multimedia contentincluding a representation of an item offered for sale by a merchant.For example, a content component of the payment processing serviceprovider system may be configured to receive the multimedia contentand/or to retrieve the multimedia content. For example, a merchantsystem or other system may push the multimedia content to the paymentprocessing service provider system without a specific request for suchcontent. In other examples, the content component may query one or moreother systems for the multimedia content. In still other examples, thecontent component may receive an indication that multimedia contentassociated with a given merchant has been requested to be output on auser device associated with a customer. In these examples, the contentcomponent may query for an instance of the multimedia content andperform the techniques for generating an interactive element overlay,such as prior to the multimedia content being output on the user device.

At block 904, the process 900 may include identifying, through one ormore recognition techniques, the item in the multimedia content. Forexample, an item-recognition component may utilize image data of themultimedia content to identify items depicted in the image data. Suchanalysis may include the use of computer vision techniques to identifythe presence of objects in given image data and then to identify theobjects themselves and/or object categories to which the objects belong(e.g., shirt, pants, hat, watch, etc.). Additional details on the use ofcomputer vision techniques is provided below. Additionally, oralternatively, when the multimedia content includes user speech, speechrecognition and natural language understanding techniques may beutilized to identify the speech, generate text data representing thespeech, and then determine an intent and/or purpose of the speech. Bydoing so, the item-recognition component may identify items referencedin the multimedia content as well as, in examples, attributes of theitems (e.g., color, size, brand, etc.). Additionally, or alternatively,the item-recognition component may utilize metadata associated with themultimedia content and/or the merchant from which the content wasprovided to identify the items. For example, the merchant system mayprovide metadata indicating the items referenced in the multimediacontent. In examples, one or more other users may have commented on orotherwise provided information in relation to the multimedia content. Inthese and other examples, some or all of this data may be utilized bythe item-recognition component to identify the items and/or attributesassociated with the items referenced in the multimedia content.

At block 906, the process 900 may include determining, based at least inpart on inventory data associated with the merchant, identifyinginformation associated with the item. For example, an item-informationcomponent of the payment processing service provider system may receiveand/or determine item information associated with the items referencedin the multimedia content. For example, the merchant system may providedata indicating information associated with the referenced items. Thatinformation may include information related to attributes of the item(e.g., sizes, colors, brands, item types, item options, etc.).Additionally, or alternatively, the item-information component may queryone or more systems for item information. By way of example, theitem-information component may query the merchant system for inventorydata indicating a current inventory of the item(s), such as at a timethat the multimedia content is being output. In examples, the merchantsystem may return the inventory data and that inventory data may beutilized to inform a customer of a current inventory of items asavailable from the merchant. In other examples, such as where theinventory data indicates the item is out of stock and/or when userpreferences indicate that a different merchant is preferred by thecustomer, an indication of the current inventory of one or more othermerchants may be retrieved and displayed on the user device. Theitem-information component may also receive item information from theitem-recognition component, such as when the item-recognition componentdetermines one or more attributes of the item utilizing the techniquesdescribed herein. Some use cases may include modification when a user ispurchasing antiques, jewelry, or gardening supplies. For example, whenpurchasing antiques, modifications to the display of information may bemade based at least in part on the item being identified as unique orrare. In the jewelry example, item information associated with qualitycontrol of the item and/or certification of the item may be provided. Inthe gardening example, preferences based on location of items (such asfor seasonal needs) may be utilized to determine what information willbe displayed. In addition to item information display modification, themodifications based on item use case may include modifications towidgets that are displayed, types of interactions that are available,and/or modifications during the ordering process.

At block 908, the process 900 may include associating the identifyinginformation with an interactive element to be presented in associationwith the multimedia content, wherein the interactive element, whenselected, causes a user device to display a graphical user interfaceconfigured to allow a customer to purchase the item. For example, aninteractive-element generator of the payment processing service providersystem may be configured to utilize the data received and/or determinedby the item-recognition component and/or the item-information componentto generate data representing an interactive element. The interactiveelement may be configured such that, when the multimedia content isoutput on the user device, the interactive element is also presented,such as in the form of an overlay or other superimposition technique.The interactive element may be, in examples, specific to the multimediacontent, the items referenced therein, the item attributes, and/or userpreferences. For example, utilizing the data received and/or determinedas described herein, the interactive-element generator may determine atype of interactive element to generate. Interactive element types mayinclude, for example, a selectable link, a QR code, an indicator thatvoice input for selection of the interactive element is enabled, anindicator that gesture input for selection of the interactive element isenabled, etc. It should be understood that while several examples ofelement types have been provided herein, this disclosure includes anyelement type that allows user input to be received. Determining the typeof interactive element to associate with given multimedia content may bebased at least in part on a device type of the user device. For example,if the device type indicates that the device includes a camera,gesture-based interactive elements may be utilized, or if the devicetype indicates the device does not include a touchscreen, theinteractive element may be configured to accept user input other thantouchscreen input. Additionally, or alternatively, purchase historyassociated with a user profile that is being utilized to view themultimedia content may be utilized to determine past user input types,and that information may be utilized to determine the type ofinteractive element to be generated. When QR codes are utilized, the QRcodes may have actions embedded therein, such as payment links,functionality to allow for adding of items to virtual carts, additionalinformation regarding the item as issue, etc. The actions, which may beconfigured by the payment processing service provider system and/or themerchant, are embedded into the interactive element at issue.Interactive elements may include any selectable element that may bepresented to a user and be configured to receive user input indicatingselection. E-commerce tags can be a type of interactive element and maybe selectable portions associated with multimedia content that mayinclude e-commerce specific actions, such as purchasing actions, iteminformation actions, and/or other shopping related actions, for example.

In addition to the type of interactive element, the interactive-elementgenerator may be configured to determine one or more other aspectsassociated with the interactive element, such as when to display theinteractive element with respect to the multimedia content, where todisplay the interactive element with respect to a viewable window of auser device, an amount and/or type of item detail to display, and/orfunctionality that occurs when the interactive element is selected, forexample. By way of example, the interactive-element generator maydetermine when to display the interactive element based at least in parton data indicating when the item begins being referenced in themultimedia content and when the item ceases being referenced. Forexample, given content may be two minutes in length, but the item maynot start being referenced until the 30-second mark and then ceasesbeing referenced at the 1-minute mark. Utilizing the item-recognitiondata described herein, the interactive-element generator may generatethe interactive element to be configured to be displayed only during thetime frame when the item is referenced. With respect to determiningwhere to display the interactive element, the interactive-elementgenerator may determine, utilizing the item-recognition data, a relativelocation of the item as depicted in the multimedia content with respectto a viewable window of the user device. For example, theitem-recognition data may indicate a location of the object asidentified in the image data, and the interactive element may begenerated such that, when displayed, the interactive element may bepositioned near the object but not, in examples, over top of the object.This will allow a user to see both the object and the interactiveelement, and perceive that the object and the interactive element areassociated with each other, while the multimedia content is output.

At block 910, the process 900 may include overlaying, for customerinteraction, the interactive element onto a portion of the multimediacontent. For example, the interactive element may be displayed, such asby way of an overlay, and may be displayed while the item is beingreferenced by the multimedia content, when output.

At block 912, the process 900 may include determining whether input datahas been received indicating customer interaction with the interactiveelement. For example, the input data may correspond to user input to thecustomer device indicating selection of the interactive element.

At block 914, the process 900 may include, when the input data has beenreceived, causing, based at least in part on the input data, the userdevice of the customer to display the graphical user interfaceconfigured to allow the customer to purchase the item. For example, acommand generator of the payment processing service provider system maybe configured to generate commands that, among other things, causedevices such as user devices to perform actions. For example, thecommand generator may also generate a command to cause the user deviceto display a purchasing user interface in response to selection of oneor more interactive elements. The command generator may also generate acommand to cause the user device to display information in the userinterface. For example, one or more user-input fields of the purchasinguser interface may be prepopulated based at least in part on some or allof the data discussed herein. For example, data from the user profilemay be utilized to prepopulate attributes and/or options associated witha selected item. Additionally, or alternatively, the item informationdetermined from the multimedia content may be utilized to prepopulatethe item attributes. Additionally, payment information from pasttransactions associated with the user profile may be utilized toprepopulate payment options, delivery addresses, etc. on the purchasinguser interface. In this way, upon selection of the interactiveelement(s), the purchasing user interface may be automatically displayedand prepopulated with item and payment instrument information such thata user may need to only confirm the purchase without providing anyadditional input to acquire the item.

At block 916, the process 900 may end if the input data is not received.

Additionally, or alternatively, the process 900 may include receivinginventory data from a system associated with the merchant, the inventorydata indicating a current inventory of the item available for purchasefrom the merchant. The process 900 may also include associating anumerical representation of the current inventory with the interactiveelement, wherein overlaying the interactive element onto the portion ofthe multimedia content includes overlaying the numerical representationof the current inventory onto the portion of the multimedia content.

Additionally, or alternatively, the process 900 may include determiningto refrain from causing display of the graphical user interface untilthe multimedia content ceases. The process 900 may also include storingfirst data indicating that the interactive element was selected. Theprocess 900 may also include receiving additional input data indicatingcustomer interaction with an additional interactive element associatedwith an additional item in the multimedia content and storing seconddata indicating that the additional interactive element was selected.The process 900 may also include, based at least in part on themultimedia content ceasing and utilizing the first data and the seconddata, causing the graphical user interface to display purchaseinformation for the item and the additional item.

Additionally, or alternatively, the process 900 may include determiningan attribute of the item as presented in the multimedia content, theattribute including a selectable option associated with the item. Theprocess 900 may also include receiving, from a system associated withthe merchant, cost data indicating a cost of the item having theattribute. The process 900 may also include causing the graphical userinterface to include the cost and causing the graphical user interfaceto include the attribute as prepopulated information in a user-inputfield.

FIG. 10 illustrates another example process 1000 for electronic commerce(“e-commerce”) tags in multimedia content. The order in which theoperations or steps are described is not intended to be construed as alimitation, and any number of the described operations may be combinedin any order and/or in parallel to implement process 1000.

At block 1002, the process 1000 may include receiving multimedia contentincluding a representation of an item. For example, a content componentof the payment processing service provider system may be configured toreceive the multimedia content and/or to retrieve the multimediacontent. For example, a merchant system or other system may push themultimedia content to the payment processing service provider systemwithout a specific request for such content. In other examples, thecontent component may query one or more other systems for the multimediacontent. In still other examples, the content component may receive anindication that multimedia content associated with a given merchant hasbeen requested to be output on a user device associated with a customer.In these examples, the content component may query for an instance ofthe multimedia content and perform the techniques for generating aninteractive element overlay, such as prior to the multimedia contentbeing output on the user device.

At block 1004, the process 1000 may include identifying the item in themultimedia content. For example, an item-recognition component mayutilize image data of the multimedia content to identify items depictedin the image data. Such analysis may include the use of computer visiontechniques to identify the presence of objects in given image data andthen to identify the objects themselves and/or object categories towhich the objects belong (e.g., shirt, pants, hat, watch, etc.).Additional details on the use of computer vision techniques is providedbelow. Additionally, or alternatively, when the multimedia contentincludes user speech, speech recognition and natural languageunderstanding techniques may be utilized to identify the speech,generate text data representing the speech, and then determine an intentand/or purpose of the speech. By doing so, the item-recognitioncomponent may identify items referenced in the multimedia content aswell as, in examples, attributes of the items (e.g., color, size, brand,etc.). Additionally, or alternatively, the item-recognition componentmay utilize metadata associated with the multimedia content and/or themerchant from which the content was provided to identify the items. Forexample, the merchant system may provide metadata indicating the itemsreferenced in the multimedia content. In examples, one or more otherusers may have commented on or otherwise provided information inrelation to the multimedia content. In these and other examples, some orall of this data may be utilized by the item-recognition component toidentify the items and/or attributes associated with the itemsreferenced in the multimedia content.

At block 1006, the process 1000 may include determining identifyinginformation associated with the item. For example, an item-informationcomponent of the payment processing service provider system may receiveand/or determine item information associated with the items referencedin the multimedia content. For example, the merchant system may providedata indicating information associated with the referenced items. Thatinformation may include information related to attributes of the item(e.g., sizes, colors, brands, item types, item options, etc.).Additionally, or alternatively, the item-information component may queryone or more systems for item information. By way of example, theitem-information component may query the merchant system for inventorydata indicating a current inventory of the item(s), such as at a timethat the multimedia content is being output. In examples, the merchantsystem may return the inventory data and that inventory data may beutilized to inform a customer of a current inventory of items asavailable from the merchant. In other examples, such as where theinventory data indicates the item is out of stock and/or when userpreferences indicate that a different merchant is preferred by thecustomer, an indication of the current inventory of one or more othermerchants may be retrieved and displayed on the user device. Theitem-information component may also receive item information from theitem-recognition component, such as when the item-recognition componentdetermines one or more attributes of the item utilizing the techniquesdescribed herein.

At block 1008, the process 1000 may include associating the identifyinginformation with an interactive element to be presented in associationwith the multimedia content, wherein the interactive element isselectable by a user of a user device. For example, aninteractive-element generator of the payment processing service providersystem may be configured to utilize the data received and/or determinedby the item-recognition component and/or the item-information componentto generate data representing an interactive element. The interactiveelement may be configured such that, when the multimedia content isoutput on the user device, the interactive element is also presented,such as in the form of an overlay. The interactive element may be, inexamples, specific to the multimedia content, the items referencedtherein, the item attributes, and/or user preferences. For example,utilizing the data received and/or determined as described herein, theinteractive-element generator may determine a type of interactiveelement to generate. Interactive element types may include, for example,a selectable link, a quick response code (“QR code”), an indicator thatvoice input for selection of the interactive element is enabled, anindicator that gesture input for selection of the interactive element isenabled, etc. It should be understood that while several examples ofelement types have been provided herein, this disclosure includes anyelement type that allows user input to be received. Determining the typeof interactive element to associate with given multimedia content may bebased at least in part on a device type of the user device. For example,if the device type indicates that the device includes a camera,gesture-based interactive elements may be utilized, or if the devicetype indicates the device does not include a touchscreen, theinteractive element may be configured to accept user input other thantouchscreen input. Additionally, or alternatively, purchase historyassociated with a user profile that is being utilized to view themultimedia content may be utilized to determine past user input types,and that information may be utilized to determine the type ofinteractive element to be generated. Interactive elements may includeany selectable element that may be presented to a user and be configuredto receive user input indicating selection. E-commerce tags can be atype of interactive element and may be selectable portions associatedwith multimedia content that may include e-commerce specific actions,such as purchasing actions, item information actions, and/or othershopping related actions, for example.

In addition to the type of interactive element, the interactive-elementgenerator may be configured to determine one or more other aspectsassociated with the interactive element, such as when to display theinteractive element with respect to the multimedia content, where todisplay the interactive element with respect to a viewable window of auser device, an amount and/or type of item detail to display, and/orfunctionality that occurs when the interactive element is selected, forexample. By way of example, the interactive-element generator maydetermine when to display the interactive element based at least in parton data indicating when the item begins being referenced in themultimedia content and when the item ceases being referenced. Forexample, given content may be two minutes in length, but the item maynot start being referenced until the 30-second mark and then ceasesbeing referenced at the 1-minute mark. Utilizing the item-recognitiondata described herein, the interactive-element generator may generatethe interactive element to be configured to be displayed only during thetime frame when the item is referenced. With respect to determiningwhere to display the interactive element, the interactive-elementgenerator may determine, utilizing the item-recognition data, a relativelocation of the item as depicted in the multimedia content with respectto a viewable window of the user device. For example, theitem-recognition data may indicate a location of the object asidentified in the image data, and the interactive element may begenerated such that, when displayed, the interactive element may bepositioned near the object but not, in examples, over top of the object.This will allow a user to see both the object and the interactiveelement, and perceive that the object and the interactive element areassociated with each other, while the multimedia content is output. Itshould be understood that the interactive elements generated andutilized herein may be e-commerce tags. E-commerce tags can beselectable portions associated with multimedia content that may includee-commerce specific actions, such as purchasing actions, iteminformation actions, and/or other shopping related actions, for example.Interactive elements may include any selectable element that may bepresented to a user and be configured to receive user input indicatingselection.

At block 1010, the process 1000 may include associating the interactiveelement with a portion of the multimedia content. For example, theitem-recognition component and/or metadata may indicate when the item inquestion starts being referenced and when the item stops beingreferenced with respect to the multimedia content. The interactiveelement may be associated with that time frame.

At block 1012, the process 1000 may include determining whether inputdata has been received from the user device indicating user interactionwith the interactive element. For example, the input data may correspondto user input to the customer device indicating selection of theinteractive element.

At block 1014, the process 1000 may include, when the input data isreceived, causing, based at least in part on the input data, the userdevice to display the graphical user interface configured to allow theuser to interact with the item. For example, a command generator of thepayment processing service provider system may be configured to generatecommands that, among other things, cause devices such as user devices toperform actions. For example, the command generator may also generate acommand to cause the user device to display a purchasing user interfacein response to selection of one or more interactive elements. Thecommand generator may also generate a command to cause the user deviceto display information in the user interface. For example, one or moreuser-input fields of the purchasing user interface may be prepopulatedbased at least in part on some or all of the data discussed herein. Forexample, data from the user profile may be utilized to prepopulateattributes and/or options associated with a selected item. Additionally,or alternatively, the item information determined from the multimediacontent may be utilized to prepopulate the item attributes.Additionally, payment information from past transactions associated withthe user profile may be utilized to prepopulate payment options,delivery addresses, etc. on the purchasing user interface. In this way,upon selection of the interactive element(s), the purchasing userinterface may be automatically displayed and prepopulated with item andpayment instrument information such that a user may need to only confirmthe purchase without providing any additional input to acquire the item.

At block 1016, the process 1000 may end if input data is not received.

Additionally, or alternatively, the process 1000 may include determininga first time indicator for when the representation of the item isreferenced in the multimedia content. The process 1000 may also includedetermining a second time indicator for when the representation of theitem ceases being referenced in the multimedia content. The process 1000may also include causing the interactive element to be displayed inassociation with the multimedia content from the first time indicator tothe second time indicator as the multimedia content is output via theuser device.

Additionally, or alternatively, the process 1000 may include determininga location, on a viewable window of the user device, where the item ispresented during output of the multimedia content by the user device.The process 1000 may also include causing the interactive element to bepresented in association with the location as the multimedia content isoutput via the user device.

Additionally, or alternatively, the process 1000 may include determininga portion of the multimedia content where the representation isdisplayed. The process 1000 may also include identifying an attribute ofthe item utilizing image data from the portion of the multimediacontent. The process 1000 may also include including the attribute as atleast a portion of the identifying information.

Additionally, or alternatively, the process 1000 may include receivingtext data representing one or more comments associated with themultimedia content. The process 1000 may also include identifying, fromthe text data, one or more keywords associated with the item. Theprocess 1000 may also include modifying the identifying informationbased at least in part on the one or more keywords.

Additionally, or alternatively, the process 1000 may includedetermining, when the user input is received, that the item isunavailable from the first merchant. The process 1000 may also includeidentifying one or more second merchants from which the item isavailable. The process 1000 may also include causing the graphical userinterface to include an identifier of the one or more second merchants.

Additionally, or alternatively, the process 1000 may include receivinginventory data from a system associated with a merchant that sells theitem, the inventory data indicating a current inventory of the itemavailable for purchase from the merchant. The process 1000 may alsoinclude determining that the current inventory is less than a thresholdinventory value. The process 1000 may also include sending, based atleast in part on the current inventory being less than the thresholdinventory value, a recommendation for a substitute item that iscurrently in stock. The substitute item may include a variation on theitem depicted in the multimedia content. For example, one or moreattributes may differ such as the color, size, likeness, etc. The dataindicating the substitute may be saved in a catalog such that the sameinteractive element, when selected, directs to the substitution insteadof the original item. The catalog data may provide indications ofassociations between items, interactive elements, and substitute itemsand/or associations indicating substitute merchants.

Additionally, or alternatively, the process 1000 may include generatingfirst data representing the interactive element, the first data beingseparate from the multimedia content. The process 1000 may also includereceiving an indication that the multimedia content has been requestedto be output on the user device. The process 1000 may also include inresponse to the indication, causing the first data to be overlaid on themultimedia content while the multimedia content is output on the userdevice. In this way, even when the multimedia content is changed and/orif the multimedia content becomes inoperable, the item information andlink associated with the interactive element may be retained andutilized. In these examples, even in situations where item informationchanges, such as over time, the multimedia content and interactiveelement(s) associated therewith may be still associated with updatedinventory information. This may decrease or eliminate the need forchanges to the multimedia content itself. Additionally, metadataassociated with user interactions with interactive elements and/ormetadata associated with the items and/or interactive elementsthemselves may be separate from purchase data associated with purchasingof the items.

Additionally, or alternatively, the process 1000 may include receivingitem information about the item from a system associated with amerchant. The process 1000 may also include receiving, via the graphicaluser interface, payment information from the user device. The process1000 may also include initiating a payment transaction for purchase ofthe item utilizing the payment information and the item information.

Additionally, or alternatively, the process 1000 may include determininga geographic region associated with the user device when the user inputdata is received. The process 1000 may also include determining acurrent inventory for the item in the geographic region, wherein causingthe interactive element to be displayed includes causing an indicationof the current inventory to be displayed.

Additionally, or alternatively, the process 1000 may include analyzing,using one or more computer vision processes and prior to sending aninstance of the multimedia content to the user device, image data of themultimedia content to identify objects depicted in the multimediacontent. The process 1000 may also include generating, prior to sendingthe instance of the multimedia content, first data including theinteractive element, wherein the interactive element is based at leastin part on the objects as identified and sending, to the user device,the first data and the instance of the multimedia content.

Additionally, or alternatively, the process 1000 may include receiving,from the user device and while the multimedia content is being output,image data depicting a user of the user device making a gesture, theuser input data including the image data. The process 1000 may alsoinclude determining a movement pattern of the gesture based at least inpart on analysis of the image data. The process 1000 may also includedetermining that the movement pattern corresponds to a referencemovement pattern indicating that the user has provided input to selectthe item for purchase. The process 1000 may also include causing theaction to be performed based at least in part on the movement patterncorresponding to the reference movement pattern.

Additionally, or alternatively, the process 1000 may includedetermining, utilizing speech recognition processes performed on audiodata of the multimedia content, text data indicating a speech portion ofthe multimedia content. The process 1000 may also include identifyingone or more characteristics of the item based at least in part on thetext data. The process 1000 may also include prepopulating at least onefield of the graphical user interface with the one or morecharacteristics.

FIG. 11 illustrates an example process 1100 for customized e-commercetags in real-time multimedia content. The order in which the operationsor steps are described is not intended to be construed as a limitation,and any number of the described operations may be combined in any orderand/or in parallel to implement process 1100.

At block 1102, the process 1100 may include receiving multimedia contentincluding a representation of an item offered for sale by a merchant.For example, a content component of the payment processing serviceprovider system may be configured to receive the multimedia contentand/or to retrieve the multimedia content. For example, a merchantsystem or other system may push the multimedia content to the paymentprocessing service provider system without a specific request for suchcontent. In other examples, the content component may query one or moreother systems for the multimedia content. In still other examples, thecontent component may receive an indication that multimedia contentassociated with a given merchant has been requested to be output on auser device associated with a customer. In these examples, the contentcomponent may query for an instance of the multimedia content andperform the techniques for generating an interactive element overlay,such as prior to the multimedia content being output on the user device.

In instances where the multimedia content has been received, the process1100 may include, at block 1104, determining a user profile associatedwith a specific user device that has requested to output the multimediacontent, the user profile including a purchase history of a user. Forexample, the user device from which a request to view the multimediacontent was received may be associated with a device identifier. Thedevice identifier may be associated with a given user profile. In otherexamples, access information associated with the user profile may beutilized to log in or otherwise access a platform utilized for viewingthe multimedia content, and that access information may be utilized toidentify the user profile.

At block 1106, the process 1100 may include determining a userpreference based at least in part on the purchase history. For example,the purchase history may indicate one or more aspects of the userprofile and/or ways in which a user associated with the user profileinteracts with devices and/or platforms. By way of example, purchasehistory may indicate typical sizes that have been ordered, itemquantities, item colors, brands, item options, payment information, userinputs, user feedback, user associations with other accounts, discountinformation, loyalty membership, etc. Some or all of this informationmay be utilized to determine one or more user preferences.

At block 1108, the process 1100 may include generating identifyinginformation associated with the item, the identifying informationemphasizing details about the item that are associated with the userpreference. For example, determining the type of interactive element toassociate with given multimedia content may be based at least in part onthe user preference. The purchase history associated with the userprofile that is being utilized to view the multimedia content may beutilized to determine past user input types, and that information may beutilized to determine the type of interactive element to be generated.In addition to the type of interactive element, an interactive-elementgenerator may utilize the item information from the item-informationcomponent to determine attributes associated with the referenced item.In some examples, all of the attributes may be included in theinteractive element. However, in other examples, only a portion of theattributes may be included. For example, utilizing the historical dataassociated with the user profile, one or more user preferences may bereceived and/or determined, and those user preferences may inform theselection of which item information to include in the interactiveelement. For example, the historical data may indicate that the userassociated with the user profile at issue purchases more items with acertain degree of item detail and/or certain types of item detailprovided. By way of additional example, the historical data may be dataassociated with more than (or other than) the user profile at issue,such as historical data associated with customers of the merchant,customers of different merchants, and/or customers in general.Interactive elements may include any selectable element that may bepresented to a user and be configured to receive user input indicatingselection. E-commerce tags can be a type of interactive element and maybe selectable portions associated with multimedia content that mayinclude e-commerce specific actions, such as purchasing actions, iteminformation actions, and/or other shopping related actions, for example.

With respect to determining the functionality that will occur uponselection of the interactive element, the interactive-element generatormay receive and/or determine data indicating user preferences forselection functionality. Those user preferences may indicate that a userdesires to have a purchasing user interface displayed upon selection ofan interactive element. Those user preferences may, in other examples,indicate that a user desires to have the purchasing user interfacedisplayed only after cessation of the multimedia content or otherwise atsome time after selection of a given interactive element. This situationmay allow the user to select multiple interactive elements correspondingeach to different items before being presented with a purchasing userinterface. In these examples, the interactive elements may be configuredto be selected and then data indicating those selections may be saveduntil cessation of the multimedia content.

At block 1110, the process 1100 may include integrating the identifyinginformation with the multimedia content such that the identifyinginformation is displayed while the multimedia content is being displayedon the specific user device. For example, data representing theinteractive element may be generated and may be associated with themultimedia content such that, when the multimedia content is displayed,the interactive element is also displayed. The identifying informationmay be a part of the interactive element such that the identifyinginformation is displayed when the multimedia data is displayed. In otherexamples, the identifying information may not be included in theinteractive element but may be associated with the interactive elementsuch that, when a user interacts with the interactive element, theidentifying information is displayed for the user.

Additionally, or alternatively, the process 1100 may include receivingtransaction data associated with past transactions of the user profile.The process 1100 may also include utilizing the transaction data forprocessing past transactions and determining the purchase history basedat least in part on the transaction data. The process 1100 may alsoinclude determining one or more attributes associated with the pasttransactions, the one or more attributes indicating purchasing trends ofthe user profile. The process 1100 may also include determining anattribute about the item that corresponds to the one or more attributes,wherein generating the identifying information emphasizing the detailsabout the item includes emphasizing the attribute about the item thatcorresponds to the one or more attributes.

Additionally, or alternatively, the process 1100 may include determiningthat historical user input to the specific user device is of a specificinput type, the specific input type selected from at least one of atouchscreen input type, a click input type, a quick response code inputtype, an audible input type, or a gesture input type, wherein the userpreference includes the specific input type. The process 1100 may alsoinclude causing the identifying information to be presented as aninteractive element configured to receive the specific input type.

Additionally, or alternatively, the process 1100 may include determininga degree of item detail associated with items purchased in pasttransactions associated with the user profile, wherein the userpreference indicates the degree of item detail. The process 1100 mayalso include causing the identifying information to be displayed on thespecific user device, the identifying information including the degreeof item detail.

Additionally, or alternatively, the process 1100 may includedetermining, from the purchase history, one or more item categories ofitems previously purchased in association with the user profile, theuser preference indicating the one or more item categories. The process1100 may also include determining that the item corresponds to at leastone of the one or more item categories, wherein generating theidentifying information is based at least in part on determining thatthe item corresponds to the at least one of the one or more itemcategories.

Additionally, or alternatively, the process 1100 may includedetermining, from the purchase history of the user profile, an amount oftime that a selectable element has been displayed previously prior toreceiving user input indicating selection of the selectable element topurchase items. The process 1100 may also include causing an interactiveelement to be displayed for at least the historical amount of time whilethe multimedia content is output.

Additionally, or alternatively, the process 1100 may includedetermining, from the purchase history of the user profile, paymentinstrument information utilized in past payment transactions. Theprocess 1100 may also include receiving input data indicating selectionof the item to purchase and causing, based at least in part on the inputdata, a user device to display a graphical user interface configured toallow the user to purchase the item, the graphical user interfaceincluding an input field with the payment instrument informationprepopulated.

Additionally, or alternatively, the process 1100 may include receivingfirst data corresponding to the purchase history from one or moremerchants, the purchase history indicating items associated with pasttransactions of the user profile and determining one or more attributesassociated with the items. The process 1100 may also include receivinginput data indicating selection of the item to purchase and causing,based at least in part on the input data, a user device to display agraphical user interface configured to allow the user to purchase theitem, the graphical user interface including an input field with one ormore item-related options prepopulated based at least in part on the oneor more attributes.

Additionally, or alternatively, the process 1100 may includedetermining, from the purchase history of the user profile, merchantsthat have provided items to a user associated with the user profile. Theprocess 1100 may also include determining that a first quantity oftransactions with a first merchant of the merchants exceeds a secondquantity of transactions with a second merchant of the merchants. Theprocess 1100 may also include determining that the first merchant offersthe item for sale and determining that the second merchant offers theitem for sale, and wherein the identifying information includes anidentifier of the first merchant instead of the second merchant based atleast in part on the first quantity of transactions exceeding the secondquantity of transactions.

Additionally, or alternatively, the process 1100 may include receiving,from a user device associated with the user profile, feedback dataindicating interaction of a user with the identifying information. Theprocess 1100 may also include modifying the user preference based atleast in part on the feedback data and causing modified identifyinginformation to be displayed on the user device based on the userpreference as modified.

Additionally, or alternatively, the process 1100 may include determiningthat historical user input associated with the user profile indicatesthat multiple items represented in other multimedia data are purchasedtogether. The process 1100 may also include, based at least in part ondetermining that the multiple items are purchased together, refrainingfrom causing display of a graphical user interface configured to allowpurchasing of the item until the multimedia content ceases being output.The process 1100 may also include, in response to the multimedia contentceasing being output, causing display of the graphical user interface.

Additionally, or alternatively, the process 1100 may include determininga device type of a device being utilized in association with the userprofile to output the multimedia content. The process 1100 may alsoinclude determining one or more user input types that the device iscapable of accepting based at least in part on the device type. Theprocess 1100 may also include generating an interactive elementincluding the identifying information, the interactive elementconfigured to be displayed when the multimedia content is displayed, andthe interactive element configured to accept the one or more user inputtypes.

Additionally, or alternatively, the process 1100 may includedetermining, from the purchase history, that the user profile has beeninvolved in at least a threshold number of past purchases where itemswere purchased from interacting with other multimedia content. Theprocess 1100 may also include determining a discount value to associatewith the user profile based at least in part on the user profile beinginvolved in the at least the threshold number of past purchases. Theprocess 1100 may also include, in response to user input to select theitem for purchasing, causing a graphical user interface to be displayed,the graphical user interface indicating that the discount value is to beapplied to a payment transaction for the item.

Additionally, or alternatively, the process 1100 may include storingdata indicating historical interaction with other multimedia content topurchase items and determining one or more characteristics of the othermultimedia content. The process 1100 may also include identifyingcontent that has not been viewed in association with the user profileand that is associated with the one or more characteristics of the othermultimedia content. The process 1100 may also include sending, to a userdevice associated with the user profile, a recommendation for viewingthe content.

FIG. 12 illustrates another example sequence diagram showing a process1200 for customized e-commerce tags in real-time multimedia content. Theorder in which the operations or steps are described is not intended tobe construed as a limitation, and any number of the described operationsmay be combined in any order and/or in parallel to implement process1200.

At block 1202, the process 1200 may include the user device 102 sendinguser input data indicating selection of an interactive element duringdisplay of multimedia content. For example, an interactive element maybe displayed on the user device 102 while the user device 102 isoutputting multimedia content including an item associated with theinteractive element. A user of the user device 102 may select theinteractive element, and user input data indicating that selection maybe sent to the payment processing service provider system 104.

At block 1204, the process 1200 may include the payment processingservice provider system 104 sending a query for inventory data and/oritem data to a merchant device 106. For example, the payment processingservice provider system 104 may inquire as to the current inventory ofthe item associated with the interactive element and/or for informationassociated with the item, such as sizes that are currently available,colors, types, options, in-stock status, etc.

At block 1206, the process 1200 may include the merchant device 106sending the requested inventory data and/or item data to the paymentprocessing service provider system 104. The inventory data and/or itemdata may be utilized to determine whether the item and/or the itemhaving particular attributes (such as color and size) are currently instock and available from the merchant at issue. In situations where theinventory data and/or item data indicates that the item is availablewith desired item attributes, an indication of the inventory countand/or some or all of the item information may be displayed on the userdevice 102. In instances where the inventory data and/or item dataindicates that the item is not available from the merchant, additionaloperations may be performed to provide the user with other informationon other items and/or items available from other merchants.

At block 1208, the process 1200 may include the payment processingservice provider system 104 sending a query for inventory data and/oritem data to one or more other merchant devices 1250. The query may besimilar to the query described with respect to block 1204, above.

At block 1210, the process 1200 may include the one or more othermerchant devices sending the requested inventory data and/or item datato the payment processing service provider system 104. The inventorydata and/or item data may be sent in a similar manner as the inventorydata and/or item data sent with respect to block 1206.

At block 1212, the process 1200 may include the payment processingservice provider system 104 sending a command to the user device 102 todisplay inventory information and/or substitute merchant information.For example, one or more indicators may be displayed. Those indicatorsmay identify other merchants from which the item is available, inventorydata from the other merchants, item options available from the othermerchants, pricing information, etc.

FIG. 13 illustrates an example environment 1300. The environment 1300includes server computing device(s) 1302 that can communicate over anetwork 1304 with user devices 1306 (which, in some examples can bemerchant devices 1308 (individually, 1308(A)-1308(N))) and/or servercomputing device(s) 1310 associated with third-party serviceprovider(s). The server computing device(s) 1302 can be associated witha service provider 1312 that can provide one or more services for thebenefit of users 1314, as described below. Actions attributed to theservice provider 1312 can be performed by the server computing device(s)1302.

In at least one example, the service provider 1312 can correspond to thepayment processing service provider described above. In at least oneexample, the server computing device(s) 1302 can correspond to theserver(s) 102 and the network(s) 1304 can correspond to the network(s)108 described above with reference to FIG. 1 . In at least one example,the multimedia content service providers described above with referenceto FIG. 1 can be associated with the server computing device(s) 1310associated with third-party service provider(s).

The environment 1300 can facilitate the generation and use ofinteractive elements in association with multimedia content. Asdescribed above, a content provider such as a merchant may post orotherwise provide multimedia content. Such content can depict and/ordiscuss (e.g., reference) item(s) (e.g., goods and/or services). In someexamples, the content can be associated with an intent to sell an itemdepicted in the content (e.g., text associated with an image indicatingthat the user is looking to sell an item depicted in the content). Inother examples, the content may not be associated with an intent to sell(e.g., no explicit or implicit indication that the user desires to sellanything depicted in the content). The service provider 1312 canidentify an item referenced in the multimedia content and generate aninteractive element to be overlaid on the content while output. Theinteractive element may be selectable and may, when selected, cause apurchasing user interface to be presented.

In at least one example, techniques performed by the environment 1300can alleviate the need for users interested in selling via platformsthat output multimedia content to perform any action that they usuallywouldn't perform prior to posting content to the platforms. That is,users interested in selling via such platforms can simply post contentto one or more platforms and techniques described herein are directed tocreating selling opportunities and facilitating transactions based onsuch content.

As described above, components of the environment 1300 can create aselling opportunity even when one did not originally exist (e.g., at thetime the content was posted). That is, if a user posts content withoutan interactive element to allows for purchasing, the service provider1312 can nevertheless create an interactive element on the fly and causethat element to be displayed along with the multimedia content to allowa customer to purchase an item referenced in the content.

As described above, users of platforms (e.g., websites, applications,and other network-based communication tools provided by serviceproviders) leverage tools for online commerce (“ecommerce”). However,current technology has limitations, as described above. In someexamples, a user interested in purchasing an item posted via such aplatform is required to follow-up with the merchant via anothercommunication tool (e.g., email, text message, private message, etc.) tocoordinate the purchase. Such systems introduce unnecessary lag due toresponse times associated with the users. Furthermore, the currentinfrastructure does not allow automatic filtering of the users and putsthe onus on the merchant on whether or not to initiate a conversationwith the interested users, engage in a financial transaction with them,and so on. In other examples, a user is directed to a webpage (usuallydifferent from the webpage or platform where the interaction originated)where the user is then required to add an item to a virtual cart andprovide payment data to complete an online transaction. As such, aplatform has to establish communication interfaces between disparateplatforms, for example between a content platform (that allowsinteraction between two users) and a payment platform (that facilitatespayment transactions). These communication interfaces have to meetsecurity protocols to allow secure communications, such as exchange offinancial data. Existing technology also introduces friction when usersintend to purchase items via content-providing platforms. That is,users—both buyers and merchants—are required to perform multiple actionsto facilitate a transaction, which can include multiple exchanges ofcommunications, multiple clicks through multiple web pages, interactionsor sign-ups with multiple platforms, or the like. As such, currenttechnology is inefficient and is not user friendly. The environment 1300described herein enables frictionless (or near-frictionless)transactions via interactions with the multimedia content. Thus,techniques described herein offer improvements to existing technology.

The environment 1300 can include a plurality of user devices 1306, asdescribed above. Each one of the plurality of user devices 1306 can beany type of computing device such as a tablet computing device, a smartphone or mobile communication device, a laptop, a netbook or otherportable computer or semi-portable computer, a desktop computing device,a terminal computing device or other semi-stationary or stationarycomputing device, a dedicated device, a wearable computing device orother body-mounted computing device, an augmented reality device, avirtual reality device, an Internet of Things (IoT) device, etc. In someexamples, individual ones of the user devices can be operable by users1314. The users 1314 can be referred to as buyers, customers, sellers,merchants, borrowers, employees, employers, payors, payees, couriers andso on. The users 1314 can interact with the user devices 1306 via userinterfaces presented via the user devices 1306. In at least one example,a user interface can be presented via a web browser, or the like. Inother examples, a user interface can be presented via an application,such as a mobile application or desktop application, which can beprovided by the service provider 1312 or which can be an otherwisededicated application. In some examples, individual of the user devices1306 can have an instance or versioned instance of an application, whichcan be downloaded from an application store, for example, which canpresent the user interface(s) described herein. In at least one example,a user 1314 can interact with the user interface via touch input, spokeninput, or any other type of input.

In at least one example, the merchant device 104 and the buyer device102 described above in FIG. 1 can comprise user devices 1306 asdescribed herein. Similarly, the merchant and the buyer can compriseusers 1314 as used herein.

In at least one example, the users 1314 can include merchants 1316(individually, 1316(A)-1316(N)). In an example, the merchants 1316 canoperate respective merchant devices 1308, which can be user devices 1306configured for use by merchants 1316. For the purpose of thisdiscussion, a “merchant” can be any entity that offers items (e.g.,goods or services) for purchase or other means of acquisition (e.g.,rent, borrow, barter, etc.). The merchants 1316 can offer items forpurchase or other means of acquisition via brick-and-mortar stores,mobile stores (e.g., pop-up shops, food trucks, etc.), online stores,combinations of the foregoing, and so forth. In some examples, at leastsome of the merchants 1316 can be associated with a same entity but canhave different merchant locations and/or can have franchise/franchiseerelationships. In additional or alternative examples, the merchants 1316can be different merchants. That is, in at least one example, themerchant 1316(A) is a different merchant than the merchant 1316(B)and/or the merchant 1316(C).

For the purpose of this discussion, “different merchants” can refer totwo or more unrelated merchants. “Different merchants” therefore canrefer to two or more merchants that are different legal entities (e.g.,natural persons and/or corporate persons) that do not share accounting,employees, branding, etc. “Different merchants,” as used herein, havedifferent names, employer identification numbers (EIN)s, lines ofbusiness (in some examples), inventories (or at least portions thereof),and/or the like. Thus, the use of the term “different merchants” doesnot refer to a merchant with various merchant locations orfranchise/franchisee relationships. Such merchants—with various merchantlocations or franchise/franchisee relationships—can be referred to asmerchants having different merchant locations and/or different commercechannels.

Each merchant device 1308 can have an instance of a POS application 1318stored thereon. The POS application 1318 can configure the merchantdevice 1308 as a POS terminal, which enables the merchant 1316(A) tointeract with one or more buyers 1320. As described above, the users1314 can include buyers, such as the buyers 1320 shown as interactingwith the merchant 1316(A). For the purpose of this discussion, a “buyer”can be any entity that acquires items from merchants. While only twobuyers 1320 are illustrated in FIG. 13 , any number of buyers 1320 caninteract with the merchants 1316. Further, while FIG. 13 illustrates thebuyers 1320 interacting with the merchant 1316(A), the buyers 1320 caninteract with any of the merchants 1316.

In at least one example, interactions between the buyers 1320 and themerchants 1316 that involve the exchange of funds (from the buyers 1320)for items (from the merchants 1316) can be referred to as “POStransactions” and/or “transactions.” In at least one example, the POSapplication 1318 can determine transaction data associated with the POStransactions. Transaction data can include payment information, whichcan be obtained from a reader device 1322 associated with the merchantdevice 1308(A), user authentication data, purchase amount information,point-of-purchase information (e.g., item(s) purchased, date ofpurchase, time of purchase, etc.), etc. The POS application 1318 cansend transaction data to the server computing device(s) 1302.Furthermore, the POS application 1318 can present a UI to enable themerchant 1316(A) to interact with the POS application 1318 and/or theservice provider 1312 via the POS application 1318.

In at least one example, the merchant device 1308(A) can be aspecial-purpose computing device configured as a POS terminal (via theexecution of the POS application 1318). In at least one example, the POSterminal may be connected to a reader device 1322, which is capable ofaccepting a variety of payment instruments, such as credit cards, debitcards, gift cards, short-range communication-based payment instruments,and the like, as described below. In at least one example, the readerdevice 1322 can plug in to a port in the merchant device 1308(A), suchas a microphone port, a headphone port, an audio-jack, a data port, orother suitable port. In additional or alternative examples, the readerdevice 1322 can be coupled to the merchant device 1308(A) via anotherwired or wireless connection, such as via a Bluetooth®, BLE, and so on.Additional details are described below with reference to FIG. 14 . Insome examples, the reader device 1322 can read information fromalternative payment instruments including, but not limited to,wristbands and the like.

In some examples, the reader device 1322 may physically interact withpayment instruments such as magnetic stripe payment cards, EMV paymentcards, and/or short-range communication (e.g., near field communication(NFC), radio frequency identification (RFID), Bluetooth®, Bluetooth® lowenergy (BLE), etc.) payment instruments (e.g., cards or devicesconfigured for tapping). The POS terminal may provide a rich userinterface, communicate with the reader device 1322, and communicate withthe server computing device(s) 1302, which can provide, among otherservices, a payment processing service. The server computing device(s)1302 associated with the service provider 1312 can communicate withserver computing device(s) 1310, as described below. In this manner, thePOS terminal and reader device 1322 may collectively processtransaction(s) between the merchants 1316 and buyers 1320. In someexamples, POS terminals and reader devices can be configured inone-to-one pairings. In other examples, the POS terminals and readerdevices can be configured in many-to-one pairings (e.g., one POSterminal coupled to multiple reader devices or multiple POS terminalscoupled to one reader device). In some examples, there could be multiplePOS terminal(s) connected to a number of other devices, such as“secondary” terminals, e.g., back-of-the-house systems, printers,line-buster devices, POS readers, and the like, to allow for informationfrom the secondary terminal to be shared between the primary POSterminal(s) and secondary terminal(s), for example via short-rangecommunication technology. This kind of arrangement may also work in anoffline-online scenario to allow one device (e.g., secondary terminal)to continue taking user input, and synchronize data with another device(e.g., primary terminal) when the primary or secondary terminal switchesto online mode. In other examples, such data synchronization may happenperiodically or at randomly selected time intervals.

While, the POS terminal and the reader device 1322 of the POS system1324 are shown as separate devices, in additional or alternativeexamples, the POS terminal and the reader device 1322 can be part of asingle device. In some examples, the reader device 1322 can have adisplay integrated therein for presenting information to the buyers1320. In additional or alternative examples, the POS terminal can have adisplay integrated therein for presenting information to the buyers1320. POS systems, such as the POS system 1324, may be mobile, such thatPOS terminals and reader devices may process transactions in disparatelocations across the world. POS systems can be used for processingcard-present transactions and card-not-present (CNP) transactions, asdescribed below.

A card-present transaction is a transaction where both a buyer 1320 andhis or her payment instrument are physically present at the time of thetransaction. Card-present transactions may be processed by swipes, dips,taps, or any other interaction between a physical payment instrument(e.g., a card), or otherwise present payment instrument, and a readerdevice 1322 whereby the reader device 1322 is able to obtain paymentdata from the payment instrument. A swipe is a card-present transactionwhere a buyer 1320 slides a card, or other payment instrument, having amagnetic strip through a reader device 1322 that captures payment datacontained in the magnetic strip. A dip is a card-present transactionwhere a buyer 1320 inserts a payment instrument having an embeddedmicrochip (i.e., chip) into a reader device 1322 first. The dippedpayment instrument remains in the payment reader until the reader device1322 prompts the buyer 1320 to remove the card, or other paymentinstrument. While the payment instrument is in the reader device 1322,the microchip can create a one-time code which is sent from the POSsystem 1324 to the server computing device(s) 1310 (which can beassociated with third-party service providers that provide paymentservices, including but not limited to, an acquirer bank, an issuer,and/or a card payment network (e.g., Mastercard®, VISA®, etc.)) to bematched with an identical one-time code. A tap is a card-presenttransaction where a buyer 1320 may tap or hover his or her paymentinstrument (e.g., card, electronic device such as a smart phone runninga payment application, etc.) over a reader device 1322 to complete atransaction via short-range communication (e.g., NFC, RFID, Bluetooth®,BLE, etc.). Short-range communication enables the payment instrument toexchange information with the reader device 1322. A tap may also becalled a contactless payment.

A CNP transaction is a transaction where a card, or other paymentinstrument, is not physically present at the POS such that payment datais required to be manually keyed in (e.g., by a merchant, buyer, etc.),or payment data is required to be recalled from a card-on-file datastore, to complete the transaction.

The POS system 1324, the server computing device(s) 1302, and/or theserver computing device(s) 1310 may exchange payment information andtransaction data to determine whether transactions are authorized. Forexample, the POS system 1324 may provide encrypted payment data, userauthentication data, purchase amount information, point-of-purchaseinformation, etc. (collectively, transaction data) to server computingdevice(s) 1302 over the network(s) 1304. The server computing device(s)1302 may send the transaction data to the server computing device(s)1310. As described above, in at least one example, the server computingdevice(s) 1310 can be associated with third-party service providers thatprovide payment services, including but not limited to, an acquirerbank, an issuer, and/or a card payment network (e.g., Mastercard®,VISA®, etc.)

For the purpose of this discussion, the “payment service providers” canbe acquiring banks (“acquirer”), issuing banks (“issuer”), card paymentnetworks, and the like. In an example, an acquirer is a bank orfinancial institution that processes payments (e.g., credit or debitcard payments) and can assume risk on behalf of merchants(s). Anacquirer can be a registered member of a card association (e.g., Visa®,MasterCard®), and can be part of a card payment network. The acquirer(e.g., the server computing device(s) 1310 associated therewith) cansend a fund transfer request to a server computing device of a cardpayment network (e.g., Mastercard®, VISA®, etc.) to determine whetherthe transaction is authorized or deficient. In at least one example, theservice provider 1312 can serve as an acquirer and connect directly withthe card payment network.

The card payment network (e.g., the server computing device(s) 1310associated therewith) can forward the fund transfer request to anissuing bank (e.g., “issuer”). The issuer is a bank or financialinstitution that offers a financial account (e.g., credit or debit cardaccount) to a user. An issuer can issue payment cards to users and canpay acquirers for purchases made by cardholders to which the issuingbank has issued a payment card. The issuer (e.g., the server computingdevice(s) 1310 associated therewith) can make a determination as towhether the buyer has the capacity to absorb the relevant chargeassociated with the payment transaction. In at least one example, theservice provider 1312 can serve as an issuer and/or can partner with anissuer. The transaction is either approved or rejected by the issuerand/or the card payment network (e.g., the server computing device(s)1310 associated therewith), and a payment authorization message iscommunicated from the issuer to the POS device via a path opposite ofthat described above, or via an alternate path.

As described above, the server computing device(s) 1310, which can beassociated with payment service provider(s), may determine whether thetransaction is authorized based on the transaction data, as well asinformation relating to parties to the transaction (e.g., the buyer 1320and/or the merchant 1316(A)). The server computing device(s) 1310 maysend an authorization notification over the network(s) 1304 to theserver computing device(s) 1302, which may send the authorizationnotification to the POS system 1324 over the network(s) 1304 to indicatewhether the transaction is authorized. The server computing device(s)1302 may also transmit additional information such as transactionidentifiers to the POS system 1324. In one example, the server computingdevice(s) 1302 may include a merchant application and/or otherfunctional components for communicating with the POS system 1324 and/orthe server computing device(s) 1310 to authorize or declinetransactions.

Based on the authentication notification that is received by the POSsystem 1324 from server computing device(s) 1302, the merchant 1316(A)may indicate to the buyer 1320 whether the transaction has beenapproved. In some examples, approval may be indicated at the POS system1324, for example, at a display of the POS system 1324. In otherexamples, such as with a smart phone or watch operating as a short-rangecommunication payment instrument, information about the approvedtransaction may be provided to the short-range communication paymentinstrument for presentation via a display of the smart phone or watch.In some examples, additional or alternative information can additionallybe presented with the approved transaction notification including, butnot limited to, receipts, special offers, coupons, or loyalty programinformation.

As mentioned above, the service provider 1312 can provide, among otherservices, payment processing services, inventory management services,catalog management services, business banking services, financingservices, lending services, reservation management services,web-development services, payroll services, employee managementservices, appointment services, loyalty tracking services, restaurantmanagement services, order management services, fulfillment services,peer-to-peer payment services, onboarding services, identityverification (IDV) services, and so on. In some examples, the users 1314can access all of the services of the service provider 1312. In otherexamples, the users 1314 can have gradated access to the services, whichcan be based on risk tolerance, IDV outputs, subscriptions, and so on.In at least one example, access to such services can be availed to themerchants 1316 via the POS application 1318. In additional oralternative examples, each service can be associated with its own accesspoint (e.g., application, web browser, etc.).

The service provider 1312 can offer payment processing services forprocessing payments on behalf of the merchants 1316, as described above.For example, the service provider 1312 can provision payment processingsoftware, payment processing hardware and/or payment processing servicesto merchants 1316, as described above, to enable the merchants 1316 toreceive payments from the buyers 1320 when conducting POS transactionswith the buyers 1320. For instance, the service provider 1312 can enablethe merchants 1316 to receive cash payments, payment card payments,and/or electronic payments from buyers 1320 for POS transactions and theservice provider 1312 can process transactions on behalf of themerchants 1316.

As the service provider 1312 processes transactions on behalf of themerchants 1316, the service provider 1312 can maintain accounts orbalances for the merchants 1316 in one or more ledgers. For example, theservice provider 1312 can analyze transaction data received for atransaction to determine an amount of funds owed to a merchant 1316(A)for the transaction. In at least one example, such an amount can be atotal purchase price less fees charged by the service provider 1312 forproviding the payment processing services. Based on determining theamount of funds owed to the merchant 1316(A), the service provider 1312can deposit funds into an account of the merchant 1316(A). The accountcan have a stored balance, which can be managed by the service provider1312. The account can be different from a conventional bank account atleast because the stored balance is managed by a ledger of the serviceprovider 1312 and the associated funds are accessible via variouswithdrawal channels including, but not limited to, scheduled deposit,same-day deposit, instant deposit, and a linked payment instrument.

A scheduled deposit can occur when the service provider 1312 transfersfunds associated with a stored balance of the merchant 1316(A) to a bankaccount of the merchant 1316(A) that is held at a bank or otherfinancial institution (e.g., associated with the server computingdevice(s) 1310). Scheduled deposits can occur at a prearranged timeafter a POS transaction is funded, which can be a business day after thePOS transaction occurred, or sooner or later. In some examples, themerchant 1316(A) can access funds prior to a scheduled deposit. Forinstance, the merchant 1316(A) may have access to same-day deposits(e.g., wherein the service provider 1312 deposits funds from the storedbalance to a linked bank account of the merchant on a same day as POStransaction, in some examples prior to the POS transaction being funded)or instant deposits (e.g., wherein the service provider 1312 depositsfunds from the stored balance to a linked bank account of the merchanton demand, such as responsive to a request). Further, in at least oneexample, the merchant 1316(A) can have a payment instrument that islinked to the stored balance that enables the merchant to access thefunds without first transferring the funds from the account managed bythe service provider 1312 to the bank account of the merchant 1316(A).

In at least one example, the service provider 1312 may provide inventorymanagement services. That is, the service provider 1312 may provideinventory tracking and reporting. Inventory management services mayenable the merchant 1316(A) to access and manage a database storing dataassociated with a quantity of each item that the merchant 1316(A) hasavailable (i.e., an inventory). Furthermore, in at least one example,the service provider 1312 can provide catalog management services toenable the merchant 1316(A) to maintain a catalog, which can be adatabase storing data associated with items that the merchant 1316(A)has available for acquisition (i.e., catalog management services). In atleast one example, the catalog may include a plurality of data items anda data item of the plurality of data items may represent an item thatthe merchant 1361(A) has available for acquisition. The service provider1312 can offer recommendations related to pricing of the items,placement of items on the catalog, and multi-party fulfilment of theinventory.

In at least one example, the service provider 1312 can provide businessbanking services, which allow the merchant 1316(A) to track deposits(from payment processing and/or other sources of funds) into an accountof the merchant 1316(A), payroll payments from the account (e.g.,payments to employees of the merchant 1316(A)), payments to othermerchants (e.g., business-to-business) directly from the account or froma linked debit card, withdrawals made via scheduled deposit and/orinstant deposit, etc. Furthermore, the business banking services canenable the merchant 1316(A) to obtain a customized payment instrument(e.g., credit card), check how much money they are earning (e.g., viapresentation of available earned balance), understand where their moneyis going (e.g., via deposit reports (which can include a breakdown offees), spend reports, etc.), access/use earned money (e.g., viascheduled deposit, instant deposit, linked payment instrument, etc.),feel in control of their money (e.g., via management of depositschedule, deposit speed, linked instruments, etc.), etc. Moreover, thebusiness banking services can enable the merchants 1316 to visualizetheir cash flow to track their financial health, set aside money forupcoming obligations (e.g., savings), organize money around goals, etc.

In at least one example, the service provider 1312 can provide financingservices and products, such as via business loans, consumer loans, fixedterm loans, flexible term loans, and the like. In at least one example,the service provider 1312 can utilize one or more risk signals todetermine whether to extend financing offers and/or terms associatedwith such financing offers.

In at least one example, the service provider 1312 can provide financingservices for offering and/or lending a loan to a borrower that is to beused for, in some instances, financing the borrower's short-termoperational needs (e.g., a capital loan). For instance, a potentialborrower that is a merchant can obtain a capital loan via a capital loanproduct in order to finance various operational costs (e.g., rent,payroll, inventory, etc.). In at least one example, the service provider1312 can offer different types of capital loan products. For instance,in at least one example, the service provider 1312 can offer a dailyrepayment loan product, wherein a capital loan is repaid daily, forinstance, from a portion of transactions processed by the paymentprocessing service on behalf of the borrower. Additionally and/oralternatively, the service provider 1312 can offer a monthly repaymentloan product, wherein a capital loan is repaid monthly, for instance,via a debit from a bank account linked to the payment processingservice. The credit risk of the merchant may be evaluated using riskmodels that take into account factors, such as payment volume, creditrisk of similarly situated merchants, past transaction history,seasonality, credit history, and so on.

Additionally or alternatively, the service provider 1312 can providefinancing services for offering and/or lending a loan to a borrower thatis to be used for, in some instances, financing the borrower's consumerpurchase (e.g., a consumer loan). In at least one example, a borrowercan submit a request for a loan to enable the borrower to purchase anitem from a merchant, which can be one of the merchants 1316. Theservice provider 1312 can create the loan based at least in part ondetermining that the borrower purchased or intends to purchase the itemfrom the merchant. The loan can be associated with a balance based on anactual purchase price of the item and the borrower can repay the loanover time. In some examples, the borrower can repay the loan viainstallments, which can be paid via funds managed and/or maintained bythe service provider 1312 (e.g., from payments owed to the merchant frompayments processed on behalf of the merchant, funds transferred to themerchant, etc.). The service provider 1312 can offer specific financialproducts, such as payment instruments, tied specifically to the loanproducts. For example, in one implementation, the server provider 1312associates capital to a merchant or buyer's debit card, where the use ofthe debit card is defined by the terms of the loan. In some examples,the merchant may only use the debit card for making specific purchases.In other examples, the “installment” associated with the loan product iscredited directly via the payment instrument. The payment instrument isthus customized to the loan and/or the parties associated with the loan.

The service provider 1312 can provide web-development services, whichenable users 1314 who are unfamiliar with HTML, XML, JavaScript, CSS, orother web design tools to create and maintain professional andaesthetically pleasing websites. Some of these web page editingapplications allow users to build a web page and/or modify a web page(e.g., change, add, or remove content associated with a web page).Further, in addition to websites, the web-development services cancreate and maintain other online omni-channel presences, such as socialmedia posts for example. In some examples, the resulting web page(s)and/or other content items can be used for offering item(s) for sale viaan online/e-commerce platform. That is, the resulting web page(s) and/orother content items can be associated with an online store or offeringby the one or more of the merchants 1316. In at least one example, theservice provider 1312 can recommend and/or create content items tosupplement omni-channel presences of the merchants 1316. That is, if amerchant of the merchants 1316 has a web page, the service provider1312—via the web-development or other services—can recommend and/orcreate additional content items to be presented via other channel(s),such as social media, email, etc.

Furthermore, the service provider 1312 can provide payroll services toenable employers to pay employees for work performed on behalf ofemployers. In at least one example, the service provider 1312 canreceive data that includes time worked by an employee (e.g., throughimported timecards and/or POS interactions), sales made by the employee,gratuities received by the employee, and so forth. Based on such data,the service provider 1312 can make payroll payments to employee(s) onbehalf of an employer via the payroll service. For instance, the serviceprovider 1312 can facilitate the transfer of a total amount to be paidout for the payroll of an employee from the bank of the employer to thebank of the service provider 1312 to be used to make payroll payments.In at least one example, when the funds have been received at the bankof the service provider 1312, the service provider 1312 can pay theemployee, such as by check or direct deposit, often a day, a week, ormore after when the work was actually performed by the employee. Inadditional or alternative examples, the service provider 1312 can enableemployee(s) to receive payments via same-day or instant deposit based atleast in part on risk and/or reliability analyses performed by theservice provider 1312.

Moreover, in at least one example, the service provider 1312 can provideemployee management services for managing schedules of employees.Further, the service provider 1312 can provide appointment services forenabling users 1314 to set schedules for scheduling appointments and/orusers 1314 to schedule appointments.

In some examples, the service provider 1312 can provide restaurantmanagement services to enable users 1314 to make and/or managereservations, to monitor front-of-house and/or back-of-house operations,and so on. In such examples, the merchant device(s) 1308 and/or servercomputing device(s) 1302 can be configured to communicate with one ormore other computing devices, which can be located in the front-of-house(e.g., POS device(s)) and/or back-of-house (e.g., kitchen displaysystem(s) (KDS)). In at least one example, the service provider 1312 canprovide order management services and/or fulfillment services to enablerestaurants to manage open tickets, split tickets, and so on and/ormanage fulfillment services. In some examples, such services can beassociated with restaurant merchants, as described above. In additionalor alternative examples, such services can be any type of merchant.

In at least one example, the service provider 1312 can providefulfilment services, which can use couriers for delivery, whereincouriers can travel between multiple locations to provide deliveryservices, photography services, etc. Couriers can be users 1314 who cantravel between locations to perform services for a requesting user 1314(e.g., deliver items, capture images, etc.). In some examples, thecourier can receive compensation from the service provider 1312. Thecourier can employ one or more vehicles, such as automobiles, bicycles,scooters, motorcycles, buses, airplanes, helicopters, boats,skateboards, etc. Although, in other instances the courier can travel byfoot or otherwise without a vehicle. Some examples discussed hereinenable people to participate as couriers in a type of crowdsourcedservice economy. Here, essentially any person with a mobile device isable to immediately become a courier, or cease to be a courier, in acourier network that provides services as described herein. In at leastone example, the couriers can be unmanned aerial vehicles (e.g.,drones), autonomous vehicles, or any other type of vehicle capable ofreceiving instructions for traveling between locations. In someexamples, the service provider 1312 can receive requests for courierservices, automatically assign the requests to active couriers, andcommunicate dispatch instructions to couriers via user interface (e.g.,application, web browser, or other access point) presented viarespective devices 1306.

In some examples, the service provider 1312 can provide omni-channelfulfillment services. For instance, if a buyer places an order with amerchant and the merchant cannot fulfill the order because one or moreitems are out of stock or otherwise unavailable, the service provider1312 can leverage other merchants and/or sales channels that are part ofthe platform of the service provider 1312 to fulfill the buyer's order.That is, another merchant can provide the one or more items to fulfillthe order of the buyer. Furthermore, in some examples, another saleschannel (e.g., online, brick-and-mortar, etc.) can be used to fulfillthe order of the buyer.

In some examples, the service provider 1312 can enable conversationalcommerce via conversational commerce services, which can use one or moremachine learning mechanisms to analyze messages exchanged between two ormore users 1314, voice inputs into a virtual assistant or the like, todetermine intents of user(s) 1314. In some examples, the serviceprovider 1312 can utilize determined intents to automate buyer service,offer promotions, provide recommendations, or otherwise interact withbuyers in real-time. In at least one example, the service provider 1312can integrate products and services, and payment mechanisms into acommunication platform (e.g., messaging, etc.) to enable buyers to makepurchases, or otherwise transact, without having to call, email, orvisit a web page or other channel of a merchant. That is, conversationalcommerce alleviates the need for buyers to toggle back and forth betweenconversations and web pages to gather information and make purchases.

In at least one example, the service provider 1312 can provide apeer-to-peer payment service that enables peer-to-peer payments betweentwo or more users 1314. In at least one example, the service provider1312 can communicate with instances of a payment application (or otheraccess point) installed on devices 1306 configured for operation byusers 1314. In an example, an instance of the payment applicationexecuting on a first device operated by a payor can send a request tothe service provider 1312 to transfer an amount of funds (e.g., fiatcurrency or non-fiat currency such as cryptocurrency, securities, andrelated assets) from an account of the payor to an account of a payee(e.g., a peer-to-peer payment). The service provider 1312 can facilitatethe transfer and can send a notification to an instance of the paymentapplication executing on a second mobile device operated by the payeethat the transfer is in process (or has been completed). In someexamples, the service provider 1312 can send additional or alternativeinformation to the instances of the payment application (e.g., lowbalance to the payor, current balance to the payor or the payee, etc.).In some implementations, the payor and/or payee can be identifiedautomatically, e.g., based on context, proximity, prior transactionhistory, and so on. In other examples, the payee can send a request forfunds to the payor prior to the payor initiating the transfer of funds.The funds transferred can be associated with any digital currency type,including, but not limited to, cash, cryptocurrency, etc. In someembodiments, the service provider 1312 funds the request to payee onbehalf of the payor, to speed up the transfer process and compensate forany lags that may be attributed to payor's financial network.

In some implementations, the service provider 1312 can trigger thepeer-to-peer payment process through identification of a “payment proxy”having a particular syntax. For example, the syntax includes a monetarycurrency indicator prefixing one or more alphanumeric characters (e.g.,$Cash). The currency indicator operates as the tagging mechanism thatindicates to a computer system to treat the inputs as a request from thesender to transfer cash, where detection of the syntax (which includesone or more alphanumeric characters tagged by a monetary currencyindicator) triggers a transfer of cash. The currency indicator cancorrespond to various currencies including but not limited to, dollar($), euro (€), pound (£), rupee (

), yuan (¥), etc. Although use of the dollar currency indicator ($) isused herein, it is to be understood that any currency symbol couldequally be used. The peer-to-peer process can be initiated through aparticular application executing on the user devices 1306.

In some embodiments, the peer-to-peer process can be implemented withina forum context. The term “forum,” as used here, refers to a contentprovider's media channel (e.g., a social networking platform, amicroblog, a blog, video sharing platform, a music sharing platform,etc.) that enables user interaction and engagement through comments,posts, messages on electronic bulletin boards, messages on a socialnetworking platform, and/or any other types of messages. The forum canbe employed by a content provider to enable users of the forum tointeract with one another, (e.g., through creating messages, postingcomments, etc.). In some embodiments, “forum” may also refer to anapplication or webpage of an e-commerce or retail organization thatoffers products and/or services. Such websites can provide an online“form” to complete before or after the products or services are added toa virtual cart. The online form may include one or more fields toreceive user interaction and engagement. Examples include name and otheridentification of the user, shipping address of the user, etc. Some ofthese fields may be configured to receive payment information, such as apayment proxy, in lieu of other kinds of payment mechanisms, such ascredit cards, debit cards, prepaid cards, gift cards, virtual wallets,etc.

In some embodiments, the peer-to-peer process can be implemented withina communication application context, such as a messaging applicationcontext. The term “messaging application,” as used here, refers to anymessaging application that enables communication between users (e.g.,sender and recipient of a message) over a wired or wirelesscommunications network, through use of a communication message. Themessaging application can be employed by the service provider 1312. Forinstance, the service provider 1312 can offer messaging services thatprovides a communication service to users via a messaging application(e.g., chat or messaging capability). The messaging application caninclude, for example, a text messaging application for communicationbetween phones (e.g., conventional mobile telephones or smartphones), ora cross-platform instant messaging application for smartphones andphones that use the Internet for communication. The messagingapplication can be executed on a user device 1306 (e.g., mobile deviceor conventional personal computer (PC)) based on instructionstransmitted to and from the server computing device(s) 1302 (which, insuch an example can be called a “messaging server”). In some instances,the messaging application can include a payment application withmessaging capability that enables users of the payment application tocommunicate with one another. In such instances, the payment applicationcan be executed on the user device 1306 based on instructionstransmitted to and from the server computing device(s) 1302 (e.g., thepayment service discussed in this description or another payment servicethat supports payment transactions).

In at least some embodiments, the peer-to-peer process can beimplemented within a landing page context. The term “landing page,” asused here, refers to a virtual location identified by a personalizedlocation address that is dedicated to collect payments on behalf of arecipient associated with the personalized location address. Thepersonalized location address that identifies the landing page caninclude a payment proxy discussed above. The service provider 1312 cancreate the landing page to enable the recipient to conveniently receiveone or more payments from one or more senders. In some embodiments, thepersonalized location address identifying the landing page is a uniformresource locator (URL) that incorporates the payment proxy. In suchembodiments, the landing page is a web page, e.g., www.cash.me/$Cash.

In at least one example, a user 1314 may be new to the service provider1312 such that the user 1314 that has not registered (e.g., subscribedto receive access to one or more services offered by the serviceprovider) with the service provider 1312. The service provider 1312 canoffer onboarding services for registering a potential user 1314 with theservice provider 1312. In some examples, onboarding can involvepresenting various questions, prompts, and the like to a potential user1314 to obtain information that can be used to create a profile for thepotential user 1314. In at least one example, the service provider 1312can provide limited or short-term access to its services prior to, orduring, onboarding (e.g., a user of a peer-to-peer payment service cantransfer and/or receive funds prior to being fully onboarded, a merchantcan process payments prior to being fully onboarded, etc.). In at leastone example, responsive to the potential user 1314 providing allnecessary information, the potential user 1314 can be onboarded to theservice provider 1312. In such an example, any limited or short-termaccess to services of the service provider 1312 can be transitioned tomore permissive (e.g., less limited) or longer-term access to suchservices.

The service provider 1312 can be associated with IDV services, which canbe used by the service provider 1312 for compliance purposes and/or canbe offered as a service, for instance to third-party service providers(e.g., associated with the server computing device(s) 1310). That is,the service provider 1312 can offer IDV services to verify the identityof users 1314 seeking to use or using their services. Identityverification requires a buyer (or potential buyer) to provideinformation that is used by compliance departments to prove that theinformation is associated with an identity of a real person or entity.In at least one example, the service provider 1312 can perform servicesfor determining whether identifying information provided by a user 1314accurately identifies the buyer (or potential buyer) (i.e., Is the buyerwho they say they are?).

The service provider 1312 is capable of providing additional oralternative services and the services described above are offered as asampling of services. In at least one example, the service provider 1312can exchange data with the server computing device(s) 1310 associatedwith third-party service providers. Such third-party service providerscan provide information that enables the service provider 1312 toprovide services, such as those described above. In additional oralternative examples, such third-party service providers can accessservices of the service provider 1312. That is, in some examples, thethird-party service providers can be subscribers, or otherwise access,services of the service provider 1312.

Techniques described herein can be configured to operate in bothreal-time/online and offline modes. “Online” modes refer to modes whendevices are capable of communicating with the service provider 1312(e.g., the server computing device(s) 1302) and/or the server computingdevice(s) 1310 via the network(s) 1304. In some examples, the merchantdevice(s) 1308 are not capable of connecting with the service provider1312 (e.g., the server computing device(s) 1302) and/or the servercomputing device(s) 1310, due to a network connectivity issue, forexample. In additional or alternative examples, the server computingdevice(s) 1302 are not capable of communicating with the servercomputing device(s) 1310 due to network connectivity issue, for example.In such examples, devices may operate in “offline” mode where at leastsome payment data is stored (e.g., on the merchant device(s) 1308)and/or the server computing device(s) 1302 until connectivity isrestored and the payment data can be transmitted to the server computingdevice(s) 1302 and/or the server computing device(s) 1310 forprocessing.

In at least one example, the service provider 1312 can be associatedwith a hub, such as an order hub, an inventory hub, a fulfillment huband so on, which can enable integration with one or more additionalservice providers (e.g., associated with the additional server computingdevice(s) 1310). In some examples, such additional service providers canoffer additional or alternative services and the service provider 1312can provide an interface or other computer-readable instructions tointegrate functionality of the service provider 1312 into the one ormore additional service providers.

Techniques described herein are directed to services provided via adistributed system of user devices 1306 that are in communication withone or more server computing devices 1302 of the service provider 1312.That is, techniques described herein are directed to a specificimplementation—or, a practical application—of utilizing a distributedsystem of user devices 1306 that are in communication with one or moreserver computing devices 1302 of the service provider 1312 to perform avariety of services, as described above. The unconventionalconfiguration of the distributed system described herein enables theserver computing device(s) 1302 that are remotely-located from end-users(e.g., users 1314) to intelligently offer services based on aggregateddata associated with the end-users, such as the users 1314 (e.g., dataassociated with multiple, different merchants and/or multiple, differentbuyers), in some examples, in near-real time. Accordingly, techniquesdescribed herein are directed to a particular arrangement of elementsthat offer technical improvements over conventional techniques forperforming payment processing services and the like. For small businessowners in particular, the business environment is typically fragmentedand relies on unrelated tools and programs, making it difficult for anowner to manually consolidate and view such data. The techniquesdescribed herein constantly or periodically monitor disparate anddistinct merchant accounts, e.g., accounts within the control of theservice provider 1312, and those outside of the control of the serviceprovider 1312, to track the business standing (payables, receivables,payroll, invoices, appointments, capital, etc.) of the merchants. Thetechniques herein provide a consolidated view of a merchant's cash flow,predict needs, preemptively offer recommendations or services, such ascapital, coupons, etc., and/or enable money movement between disparateaccounts (merchant's, another merchant's, or even payment service's) ina frictionless and transparent manner.

As described herein, artificial intelligence, machine learning, and thelike can be used to dynamically make determinations, recommendations,and the like, thereby adding intelligence and context-awareness to anotherwise one-size-fits-all scheme for providing payment processingservices and/or additional or alternative services described herein. Insome implementations, the distributed system is capable of applying theintelligence derived from an existing user base to a new user, therebymaking the onboarding experience for the new user personalized andfrictionless when compared to traditional onboarding methods. Thus,techniques described herein improve existing technological processes.

As described above, various graphical user interfaces (GUIs) can bepresented to facilitate techniques described herein. Some of thetechniques described herein are directed to user interface featurespresented via GUIs to improve interaction between users 1314 and userdevices 1306. Furthermore, such features are changed dynamically basedon the profiles of the users involved interacting with the GUIs. Assuch, techniques described herein are directed to improvements tocomputing systems.

FIG. 14 depicts an illustrative block diagram illustrating a system 1400for performing techniques described herein. The system 1400 includes auser device 1402, that communicates with server computing device(s)(e.g., server(s) 1404) via network(s) 1406 (e.g., the Internet, cablenetwork(s), cellular network(s), cloud network(s), wireless network(s)(e.g., Wi-Fi) and wired network(s), as well as close-rangecommunications such as Bluetooth®, Bluetooth® low energy (BLE), and thelike). While a single user device 1402 is illustrated, in additional oralternate examples, the system 1400 can have multiple user devices, asdescribed above with reference to FIG. 13 .

The environment 1400 can facilitate the generation and use ofinteractive elements in association with multimedia content. Asdescribed above, a content provider such as a merchant may post orotherwise provide multimedia content. Such content can depict and/ordiscuss (e.g., reference) item(s) (e.g., goods and/or services). In someexamples, the content can be associated with an intent to sell an itemdepicted in the content (e.g., text associated with an image indicatingthat the user is looking to sell an item depicted in the content). Inother examples, the content may not be associated with an intent to sell(e.g., no explicit or implicit indication that the user desires to sellanything depicted in the content). The service provider 1312 canidentify an item referenced in the multimedia content and generate aninteractive element to be overlaid on the content while output. Theinteractive element may be selectable and may, when selected, cause apurchasing user interface to be presented.

In at least one example, techniques performed by the environment 1400can alleviate the need for users interested in selling via platformsthat output multimedia content to perform any action that they usuallywouldn't perform prior to posting content to the platforms. That is,users interested in selling via such platforms can simply post contentto one or more platforms and techniques described herein are directed tocreating selling opportunities and facilitating transactions based onsuch content.

As described above, components of the environment 1400 can create aselling opportunity even when one did not originally exist (e.g., at thetime the content was posted). That is, if a user posts content withoutan interactive element to allows for purchasing, the service provider1412 can nevertheless create an interactive element on the fly and causethat element to be displayed along with the multimedia content to allowa customer to purchase an item referenced in the content.

As described above, users of platforms (e.g., websites, applications,and other network-based communication tools provided by serviceproviders) leverage tools for online commerce (“ecommerce”). However,current technology has limitations, as described above. In someexamples, a user interested in purchasing an item posted via such aplatform is required to follow-up with the merchant via anothercommunication tool (e.g., email, text message, private message, etc.) tocoordinate the purchase. Such systems introduce unnecessary lag due toresponse times associated with the users. Furthermore, the currentinfrastructure does not allow automatic filtering of the users and putsthe onus on the merchant on whether or not to initiate a conversationwith the interested users, engage in a financial transaction with them,and so on. In other examples, a user is directed to a webpage (usuallydifferent from the webpage or platform where the interaction originated)where the user is then required to add an item to a virtual cart andprovide payment data to complete an online transaction. As such, aplatform has to establish communication interfaces between disparateplatforms, for example between a content platform (that allowsinteraction between two users) and a payment platform (that facilitatespayment transactions). These communication interfaces have to meetsecurity protocols to allow secure communications, such as exchange offinancial data. Existing technology also introduces friction when usersintend to purchase items via content-providing platforms. That is,users—both buyers and merchants—are required to perform multiple actionsto facilitate a transaction, which can include multiple exchanges ofcommunications, multiple clicks through multiple web pages, interactionsor sign-ups with multiple platforms, or the like. As such, currenttechnology is inefficient and is not user friendly. The environment 1400described herein enables frictionless (or near-frictionless)transactions via interactions with the multimedia content. Thus,techniques described herein offer improvements to existing technology.

In at least one example, the user device 1402 can be any suitable typeof computing device, e.g., portable, semi-portable, semi-stationary, orstationary. Some examples of the user device 1402 can include, but arenot limited to, a tablet computing device, a smart phone or mobilecommunication device, a laptop, a netbook or other portable computer orsemi-portable computer, a desktop computing device, a terminal computingdevice or other semi-stationary or stationary computing device, adedicated device, a wearable computing device or other body-mountedcomputing device, an augmented reality device, a virtual reality device,an Internet of Things (IoT) device, etc. That is, the user device 1402can be any computing device capable of sending communications andperforming the functions according to the techniques described herein.The user device 1402 can include devices, e.g., payment card readers, orcomponents capable of accepting payments, as described below.

In the illustrated example, the user device 1402 includes one or moreprocessors 1408, one or more computer-readable media 1410, one or morecommunication interface(s) 1412, one or more input/output (I/O) devices1414, a display 1416, and sensor(s) 1418.

In at least one example, each processor 1408 can itself comprise one ormore processors or processing cores. For example, the processor(s) 1408can be implemented as one or more microprocessors, microcomputers,microcontrollers, digital signal processors, central processing units,state machines, logic circuitries, and/or any devices that manipulatesignals based on operational instructions. In some examples, theprocessor(s) 1408 can be one or more hardware processors and/or logiccircuits of any suitable type specifically programmed or configured toexecute the algorithms and processes described herein. The processor(s)1408 can be configured to fetch and execute computer-readableprocessor-executable instructions stored in the computer-readable media1410.

Depending on the configuration of the user device 1402, thecomputer-readable media 1410 can be an example of tangiblenon-transitory computer storage media and can include volatile andnonvolatile memory and/or removable and non-removable media implementedin any type of technology for storage of information such ascomputer-readable processor-executable instructions, data structures,program modules or other data. The computer-readable media 1410 caninclude, but is not limited to, RAM, ROM, EEPROM, flash memory,solid-state storage, magnetic disk storage, optical storage, and/orother computer-readable media technology. Further, in some examples, theuser device 1402 can access external storage, such as RAID storagesystems, storage arrays, network attached storage, storage areanetworks, cloud storage, or any other medium that can be used to storeinformation and that can be accessed by the processor(s) 1408 directlyor through another computing device or network. Accordingly, thecomputer-readable media 1410 can be computer storage media able to storeinstructions, modules or components that can be executed by theprocessor(s) 1408. Further, when mentioned, non-transitorycomputer-readable media exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

The computer-readable media 1410 can be used to store and maintain anynumber of functional components that are executable by the processor(s)1408. In some implementations, these functional components compriseinstructions or programs that are executable by the processor(s) 1408and that, when executed, implement operational logic for performing theactions and services attributed above to the user device 1402.Functional components stored in the computer-readable media 1410 caninclude a user interface 1420 to enable users to interact with the userdevice 1402, and thus the server(s) 1404 and/or other networked devices.In at least one example, the user interface 1420 can be presented via aweb browser, or the like. In other examples, the user interface 1420 canbe presented via an application, such as a mobile application or desktopapplication, which can be provided by a service provider 1312 associatedwith the server(s) 1404, or which can be an otherwise dedicatedapplication. In some examples, the user interface 1420 can be one of theuser interface(s) 122 described above with reference to FIG. 1 . In atleast one example, a user can interact with the user interface via touchinput, spoken input, gesture, or any other type of input. The word“input” is also used to describe “contextual” input that may not bedirectly provided by the user via the user interface 1420. For example,user's interactions with the user interface 1420 are analyzed using,e.g., natural language processing techniques, to determine context orintent of the user, which may be treated in a manner similar to “direct”user input.

Depending on the type of the user device 1402, the computer-readablemedia 1410 can also optionally include other functional components anddata, such as other modules and data 1422, which can include programs,drivers, etc., and the data used or created by the functionalcomponents. In addition, the computer-readable media 1410 can also storedata, data structures and the like, that are used by the functionalcomponents. Further, the user device 1402 can include many otherlogical, programmatic and physical components, of which those describedare merely examples that are related to the discussion herein.

In at least one example, the computer-readable media 1410 can includeadditional functional components, such as an operating system 1424 forcontrolling and managing various functions of the user device 1402 andfor enabling basic user interactions.

The communication interface(s) 1412 can include one or more interfacesand hardware components for enabling communication with various otherdevices, such as over the network(s) 1406 or directly. For example,communication interface(s) 1412 can enable communication through one ormore network(s) 1406, which can include, but are not limited any type ofnetwork known in the art, such as a local area network or a wide areanetwork, such as the Internet, and can include a wireless network, suchas a cellular network, a cloud network, a local wireless network, suchas Wi-Fi and/or close-range wireless communications, such as Bluetooth®,BLE, NFC, RFID, a wired network, or any other such network, or anycombination thereof. Accordingly, network(s) 1406 can include both wiredand/or wireless communication technologies, including Bluetooth®, BLE,Wi-Fi and cellular communication technologies, as well as wired or fiberoptic technologies. Components used for such communications can dependat least in part upon the type of network, the environment selected, orboth. Protocols for communicating over such networks are well known andwill not be discussed herein in detail.

Embodiments of the disclosure may be provided to users through a cloudcomputing infrastructure. Cloud computing refers to the provision ofscalable computing resources as a service over a network, to enableconvenient, on-demand network access to a shared pool of configurablecomputing resources that can be rapidly provisioned and released withminimal management effort or service provider interaction. Thus, cloudcomputing allows a user to access virtual computing resources (e.g.,storage, data, applications, and even complete virtualized computingsystems) in “the cloud,” without regard for the underlying physicalsystems (or locations of those systems) used to provide the computingresources.

The user device 1402 can further include one or more input/output (I/O)devices 1414. The I/O devices 1414 can include speakers, a microphone, acamera, and various user controls (e.g., buttons, a joystick, akeyboard, a keypad, etc.), a haptic output device, and so forth. The I/Odevices 1414 can also include attachments that leverage the accessories(audio-jack, USB-C, Bluetooth, etc.) to connect with the user device1402.

In at least one example, user device 1402 can include a display 1416.Depending on the type of computing device(s) used as the user device1402, the display 1416 can employ any suitable display technology. Forexample, the display 1416 can be a liquid crystal display, a plasmadisplay, a light emitting diode display, an OLED (organic light-emittingdiode) display, an electronic paper display, or any other suitable typeof display able to present digital content thereon. In at least oneexample, the display 1416 can be an augmented reality display, avirtually reality display, or any other display able to present and/orproject digital content. In some examples, the display 1416 can have atouch sensor associated with the display 1416 to provide a touchscreendisplay configured to receive touch inputs for enabling interaction witha graphic interface presented on the display 1416. Accordingly,implementations herein are not limited to any particular displaytechnology. Alternatively, in some examples, the user device 1402 maynot include the display 1416, and information can be presented by othermeans, such as aurally, hapticly, etc.

In addition, the user device 1402 can include sensor(s) 1418. Thesensor(s) 1418 can include a GPS device able to indicate locationinformation. Further, the sensor(s) 1418 can include, but are notlimited to, an accelerometer, gyroscope, compass, proximity sensor,camera, microphone, and/or a switch.

In some example, the GPS device can be used to identify a location of auser. In at least one example, the location of the user can be used bythe service provider 1312, described above, to provide one or moreservices. That is, in some examples, the service provider 1312 canimplement geofencing to provide particular services to users. As anexample, with a lending service, location can be used to confirm that astated purpose of a loan corresponds to evidence of use (e.g., Is theuser using the loan consistent with what he or she said he or she wasgoing to use it for?). Furthermore, in some examples, location can beused for payroll purposes. As an example, if a contractor completes aproject, the contractor can provide a geo-tagged image (e.g., taggedbased on location information availed by the GPS device). In someexamples, location can be used for facilitating peer-to-peer paymentsbetween nearby users 1314 and/or for sending users 1314 notificationsregarding available appointments with merchant(s) located proximate tothe users 1314. In at least one example, location can be used for takingpayments from nearby buyers when they leave a geofence, or location canbe used to initiate an action responsive to users 1314 enter abrick-and-mortar store of a merchant. Location can be used in additionalor alternative ways as well.

Additionally, the user device 1402 can include various other componentsthat are not shown, examples of which include removable storage, a powersource, such as a battery and power control unit, a barcode scanner, aprinter, a cash drawer, and so forth.

In addition, in some examples, the user device 1402 can include, beconnectable to, or otherwise be coupled to a reader device 1426, forreading payment instruments and/or identifiers associated with paymentobjects. In some examples, as described above, the reader device 1426can plug in to a port in the user device 1402, such as a microphoneport, a headphone port, an audio-jack, a data port, or other suitableport. In additional or alternative examples, the reader device 1426 canbe coupled to the user device 1402 via another wired or wirelessconnection, such as via a Bluetooth®, BLE, and so on. The reader device1426 can include a read head for reading a magnetic strip of a paymentcard, and further can include encryption technology for encrypting theinformation read from the magnetic strip. Additionally or alternatively,the reader device 1426 can be an EMV payment reader, which in someexamples, can be embedded in the user device 1402. Moreover, numerousother types of readers can be employed with the user device 1402 herein,depending on the type and configuration of the user device 1402.

The reader device 1426 may be a portable magnetic stripe card reader,optical scanner, smartcard (card with an embedded IC chip) reader (e.g.,an EMV-compliant card reader or short-range communication-enabledreader), RFID reader, or the like, configured to detect and obtain dataoff any payment instrument. Accordingly, the reader device 1426 mayinclude hardware implementation, such as slots, magnetic tracks, andrails with one or more sensors or electrical contacts to facilitatedetection and acceptance of a payment instrument. That is, the readerdevice 1426 may include hardware implementations to enable the readerdevice 1426 to interact with a payment instrument via a swipe (i.e., acard-present transaction where a buyer slides a card having a magneticstrip through a payment reader that captures payment data contained inthe magnetic strip), a dip (i.e., a card-present transaction where abuyer inserts a card having an embedded microchip (i.e., chip) into apayment reader first until the payment reader prompts the buyer toremove the card), or a tap (i.e., a card-present transaction where abuyer may tap or hover his or her electronic device such as a smartphone running a payment application over a payment reader to complete atransaction via short-range communication) to obtain payment dataassociated with a buyer. Additionally or optionally, the reader device1426 may also include a biometric sensor to receive and processbiometric characteristics and process them as payment instruments, giventhat such biometric characteristics are registered with the paymentprocessing service provider and connected to a financial account with abank server.

The reader device 1426 may include processing unit(s), computer-readablemedia, a reader chip, a transaction chip, a timer, a clock, a networkinterface, a power supply, and so on. The processing unit(s) of thereader device 1426 may execute one or more modules and/or processes tocause the reader device 1426 to perform a variety of functions, as setforth above and explained in further detail in the following disclosure.In some examples, the processing unit(s) may include a centralprocessing unit (CPU), a graphics processing unit (GPU), a CPU and aGPU, or processing units or components known in the art. Additionally,each of the processing unit(s) may possess its own local memory, whichalso may store program modules, program data, and/or one or moreoperating systems. Depending on the exact configuration and type of thereader device 1426, the computer-readable media may include volatilememory (such as RAM), non-volatile memory (such as ROM, flash memory,miniature hard drive, memory card, or the like), or some combinationthereof. In at least one example, the computer-readable media of thereader device 1426 may include at least one module for performingvarious functions as described herein.

The reader chip may perform functionalities to control the operationsand processing of the reader device 1426. That is, the reader chip mayperform functionalities to control payment interfaces (e.g., acontactless interface, a contact interface, etc.), a wirelesscommunication interface, a wired interface, a user interface (e.g., asignal condition device (FPGA)), etc. Additionally, the reader chip mayperform functionality to control the timer, which may provide a timersignal indicating an amount of time that has lapsed following aparticular event (e.g., an interaction, a power-down event, etc.).Moreover, the reader chip may perform functionality to control a clock,which may provide a clock signal indicating a time. Furthermore, thereader chip may perform functionality to control the network interface,which may interface with the network(s) 1406, as described below.

Additionally, the reader chip may perform functionality to control thepower supply. The power supply may include one or more power suppliessuch as a physical connection to AC power or a battery. Power supply mayinclude power conversion circuitry for converting AC power and creatinga plurality of DC voltages for use by components of reader device 1426.When power supply includes a battery, the battery may be charged via aphysical power connection, via inductive charging, or via any othersuitable method.

The transaction chip may perform functionalities relating to processingof payment transactions, interfacing with payment instruments,cryptography, and other payment-specific functionality. That is, thetransaction chip may access payment data associated with a paymentinstrument and may provide the payment data to a POS terminal, asdescribed above. The payment data may include, but is not limited to, aname of the buyer, an address of the buyer, a type (e.g., credit, debit,etc.) of a payment instrument, a number associated with the paymentinstrument, a verification value (e.g., PIN Verification Key Indicator(PVKI), PIN Verification Value (PVV), Card Verification Value (CVV),Card Verification Code (CVC), etc.) associated with the paymentinstrument, an expiration data associated with the payment instrument, aprimary account number (PAN) corresponding to the buyer (which may ormay not match the number associated with the payment instrument),restrictions on what types of charges/debts may be made, etc.Additionally, the transaction chip may encrypt the payment data uponreceiving the payment data.

It should be understood that in some examples, the reader chip may haveits own processing unit(s) and computer-readable media and/or thetransaction chip may have its own processing unit(s) andcomputer-readable media. In other examples, the functionalities ofreader chip and transaction chip may be embodied in a single chip or aplurality of chips, each including any suitable combination ofprocessing units and computer-readable media to collectively perform thefunctionalities of reader chip and transaction chip as described herein.

While, the user device 1402, which can be a POS terminal, and the readerdevice 1426 are shown as separate devices, in additional or alternativeexamples, the user device 1402 and the reader device 1426 can be part ofa single device, which may be a battery-operated device. In such anexample, components of both the user device 1402 and the reader device1426 may be associated with the single device. In some examples, thereader device 1426 can have a display integrated therewith, which can bein addition to (or as an alternative of) the display 1416 associatedwith the user device 1402.

The server(s) 1404 can include one or more servers or other types ofcomputing devices that can be embodied in any number of ways. Forexample, in the example of a server, the modules, other functionalcomponents, and data can be implemented on a single server, a cluster ofservers, a server farm or data center, a cloud-hosted computing service,a cloud-hosted storage service, and so forth, although other computerarchitectures can additionally or alternatively be used.

Further, while the figures illustrate the components and data of theserver(s) 1404 as being present in a single location, these componentsand data can alternatively be distributed across different computingdevices and different locations in any manner. Consequently, thefunctions can be implemented by one or more server computing devices,with the various functionality described above distributed in variousways across the different computing devices. Multiple server(s) 1404 canbe located together or separately, and organized, for example, asvirtual servers, server banks and/or server farms. The describedfunctionality can be provided by the servers of a single merchant orenterprise or can be provided by the servers and/or services of multipledifferent buyers or enterprises.

In the illustrated example, the server(s) 1404 can include one or moreprocessors 1428, one or more computer-readable media 1430, one or moreI/O devices 1432, and one or more communication interfaces 1434. Eachprocessor 1428 can be a single processing unit or a number of processingunits and can include single or multiple computing units or multipleprocessing cores. The processor(s) 1428 can be implemented as one ormore microprocessors, microcomputers, microcontrollers, digital signalprocessors, central processing units, state machines, logic circuitries,and/or any devices that manipulate signals based on operationalinstructions. For example, the processor(s) 1428 can be one or morehardware processors and/or logic circuits of any suitable typespecifically programmed or configured to execute the algorithms andprocesses described herein. The processor(s) 1428 can be configured tofetch and execute computer-readable instructions stored in thecomputer-readable media 1430, which can program the processor(s) 1428 toperform the functions described herein.

The computer-readable media 1430 can include volatile and nonvolatilememory and/or removable and non-removable media implemented in any typeof technology for storage of information, such as computer-readableinstructions, data structures, program modules, or other data. Suchcomputer-readable media 1430 can include, but is not limited to, RAM,ROM, EEPROM, flash memory or other memory technology, optical storage,solid state storage, magnetic tape, magnetic disk storage, RAID storagesystems, storage arrays, network attached storage, storage areanetworks, cloud storage, or any other medium that can be used to storethe desired information and that can be accessed by a computing device.Depending on the configuration of the server(s) 1404, thecomputer-readable media 1430 can be a type of computer-readable storagemedia and/or can be a tangible non-transitory media to the extent thatwhen mentioned, non-transitory computer-readable media exclude mediasuch as energy, carrier signals, electromagnetic waves, and signals perse.

The computer-readable media 1430 can be used to store any number offunctional components that are executable by the processor(s) 1428. Inmany implementations, these functional components comprise instructionsor programs that are executable by the processors 1428 and that, whenexecuted, specifically configure the one or more processors 1428 toperform the actions attributed above to the service provider 1312 and/orpayment processing service. Functional components stored in thecomputer-readable media 1430 can optionally include a merchant module1436, a training module 1438, and one or more other modules and data1440.

The merchant module 1436 can be configured to receive transaction datafrom POS systems, such as the POS system 1324 described above withreference to FIG. 13 . The merchant module 1436 can transmit requests(e.g., authorization, capture, settlement, etc.) to payment serviceserver computing device(s) to facilitate POS transactions betweenmerchants and buyers. The merchant module 1436 can communicate thesuccesses or failures of the POS transactions to the POS systems. Thepayment processing module 116 described above with reference to FIGS. 1and 2 can correspond to the merchant module 1436.

The training module 1438 can be configured to train models usingmachine-learning mechanisms. For example, a machine-learning mechanismcan analyze training data to train a data model that creates an output,which can be a recommendation, a score, and/or another indication.Machine-learning mechanisms can include, but are not limited tosupervised learning algorithms (e.g., artificial neural networks,Bayesian statistics, support vector machines, decision trees,classifiers, k-nearest neighbor, etc.), unsupervised learning algorithms(e.g., artificial neural networks, association rule learning,hierarchical clustering, cluster analysis, etc.), semi-supervisedlearning algorithms, deep learning algorithms, etc.), statisticalmodels, etc. In at least one example, machine-trained data models can bestored in a data store(s) associated with the user device(s) 1402 and/orthe server(s) 1404 for use at a time after the data models have beentrained (e.g., at runtime).

The one or more other modules and data 1440 can include theinteractive-element generator 138 and/or the command generator 140, thefunctionality of which is described, at least partially, above. Further,the one or more other modules and data 1440 can include programs,drivers, etc., and the data used or created by the functionalcomponents. Further, the server(s) 1404 can include many other logical,programmatic and physical components, of which those described above aremerely examples that are related to the discussion herein.

The one or more “modules” and/or “components” referenced herein may beimplemented as more modules or as fewer modules, and functions describedfor the modules may be redistributed depending on the details of theimplementation. The term “module,” as used herein, refers broadly tosoftware stored on non-transitory storage medium (e.g., volatile ornon-volatile memory for a computing device), hardware, or firmware (orany combination thereof) modules. Modules are typically functional suchthat they that may create useful data or other output using specifiedinput(s). A module may or may not be self-contained. An applicationprogram (also called an “application”) may include one or more modules,or a module may include one or more application programs that can beaccessed over a network or downloaded as software onto a device (e.g.,executable code causing the device to perform an action). An applicationprogram (also called an “application”) may include one or more modules,or a module may include one or more application programs. In additionaland/or alternative examples, the module(s) may be implemented ascomputer-readable instructions, various data structures, and so forthvia at least one processing unit to configure the computing device(s)described herein to execute instructions and to perform operations asdescribed herein.

In some examples, a module may include one or more applicationprogramming interfaces (APIs) to perform some or all of itsfunctionality (e.g., operations). In at least one example, a softwaredeveloper kit (SDK) can be provided by the service provider to allowthird-party developers to include service provider functionality and/oravail service provider services in association with their ownthird-party applications. Additionally or alternatively, in someexamples, the service provider can utilize a SDK to integratethird-party service provider functionality into its applications. Thatis, API(s) and/or SDK(s) can enable third-party developers to customizehow their respective third-party applications interact with the serviceprovider or vice versa. The API(s) 148, described above, can correspondto such.

The computer-readable media 1430 can additionally include an operatingsystem 1442 for controlling and managing various functions of theserver(s) 1404.

The communication interface(s) 1434 can include one or more interfacesand hardware components for enabling communication with various otherdevices, such as over the network(s) 1406 or directly. For example,communication interface(s) 1434 can enable communication through one ormore network(s) 1406, which can include, but are not limited any type ofnetwork known in the art, such as a local area network or a wide areanetwork, such as the Internet, and can include a wireless network, suchas a cellular network, a local wireless network, such as Wi-Fi and/orclose-range wireless communications, such as Bluetooth®, BLE, NFC, RFID,a wired network, or any other such network, or any combination thereof.Accordingly, network(s) 1402 can include both wired and/or wirelesscommunication technologies, including Bluetooth®, BLE, Wi-Fi andcellular communication technologies, as well as wired or fiber optictechnologies. Components used for such communications can depend atleast in part upon the type of network, the environment selected, orboth. Protocols for communicating over such networks are well known andwill not be discussed herein in detail.

The server(s) 1404 can further be equipped with various I/O devices1432. Such I/O devices 1432 can include a display, various userinterface controls (e.g., buttons, joystick, keyboard, mouse, touchscreen, biometric or sensory input devices, etc.), audio speakers,connection ports and so forth.

In at least one example, the system 1400 can include data store(s) 1444that can be configured to store data that is accessible, manageable, andupdatable. In some examples, the data store(s) 1444 can be integratedwith the user device 1402 and/or the server(s) 1404. In other examples,as shown in FIG. 14 , the data store(s) 1444 can be located remotelyfrom the server(s) 1404 and can be accessible to the server(s) 1404. Thedata store(s) 1444 can comprise multiple databases and/or serversconnected locally and/or remotely via the network(s) 1406. The datastore(s) 150, described above with reference to FIG. 1 , can correspondto the data store(s) 1444.

In at least one example, the data store(s) 1444 can store user profiles,which can include merchant profiles, buyer profiles, and so on.

Merchant profiles can store, or otherwise be associated with, dataassociated with merchants. For instance, a merchant profile can store,or otherwise be associated with, information about a merchant (e.g.,name of the merchant, geographic location of the merchant, operatinghours of the merchant, employee information, etc.), a merchant categoryclassification (MCC), item(s) offered for sale by the merchant, hardware(e.g., device type) used by the merchant, transaction data associatedwith the merchant (e.g., transactions conducted by the merchant, paymentdata associated with the transactions, items associated with thetransactions, descriptions of items associated with the transactions,itemized and/or total spends of each of the transactions, parties to thetransactions, dates, times, and/or locations associated with thetransactions, etc.), loan information associated with the merchant(e.g., previous loans made to the merchant, previous defaults on saidloans, etc.), risk information associated with the merchant (e.g.,indications of risk, instances of fraud, chargebacks, etc.),appointments information (e.g., previous appointments, upcoming(scheduled) appointments, timing of appointments, lengths ofappointments, etc.), payroll information (e.g., employees, payrollfrequency, payroll amounts, etc.), employee information, reservationsdata (e.g., previous reservations, upcoming (scheduled) reservations,interactions associated with such reservations, etc.), inventory data,buyer service data, etc. The merchant profile can securely store bankaccount information as provided by the merchant. Further, the merchantprofile can store payment information associated with a paymentinstrument linked to a stored balance of the merchant, such as a storedbalance maintained in a ledger by the service provider 1312.

Buyer profiles can store buyer data including, but not limited to, buyerinformation (e.g., name, phone number, address, banking information,etc.), buyer preferences (e.g., learned or buyer-specified), purchasehistory data (e.g., identifying one or more items purchased (andrespective item information), payment instruments used to purchase oneor more items, returns associated with one or more orders, statuses ofone or more orders (e.g., preparing, packaging, in transit, delivered,etc.), etc.), appointments data (e.g., previous appointments, upcoming(scheduled) appointments, timing of appointments, lengths ofappointments, etc.), payroll data (e.g., employers, payroll frequency,payroll amounts, etc.), reservations data (e.g., previous reservations,upcoming (scheduled) reservations, reservation duration, interactionsassociated with such reservations, etc.), inventory data, buyer servicedata, etc.

In at least one example, the account(s), described above with referenceto FIG. 1 , can include or be associated with the merchant profilesand/or buyer profiles described above.

Furthermore, in at least one example, the data store(s) 1444 can storeinventory database(s) and/or catalog database(s). As described above, aninventory can store data associated with a quantity of each item that amerchant has available to the merchant. The records described above canbe stored in the inventory data store. Furthermore, a catalog can storedata associated with items that a merchant has available foracquisition. The data store(s) 1444 can store additional or alternativetypes of data as described herein.

The phrases “in some examples,” “according to various examples,” “in theexamples shown,” “in one example,” “in other examples,” “variousexamples,” “some examples,” and the like generally mean the particularfeature, structure, or characteristic following the phrase is includedin at least one example of the present invention, and may be included inmore than one example of the present invention. In addition, suchphrases do not necessarily refer to the same examples or to differentexamples.

If the specification states a component or feature “can,” “may,”“could,” or “might” be included or have a characteristic, thatparticular component or feature is not required to be included or havethe characteristic.

Further, the aforementioned description is directed to devices andapplications that are related to payment technology. However, it will beunderstood, that the technology can be extended to any device andapplication. Moreover, techniques described herein can be configured tooperate irrespective of the kind of payment object reader, POS terminal,web applications, mobile applications, POS topologies, payment cards,computer networks, and environments.

Various figures included herein are flowcharts showing example methodsinvolving techniques as described herein. The methods illustrated aredescribed with reference to FIGS. 5-12 for convenience and ease ofunderstanding. However, the methods illustrated are not limited to beingperformed using components described in FIGS. 1-4D, 13, and 14 , andsuch components are not limited to performing the methods illustratedherein.

Furthermore, the methods described above are illustrated as collectionsof blocks in logical flow graphs, which represent sequences ofoperations that can be implemented in hardware, software, or acombination thereof. In the context of software, the blocks representcomputer-executable instructions stored on one or more computer-readablestorage media that, when executed by processor(s), perform the recitedoperations. Generally, computer-executable instructions includeroutines, programs, objects, components, data structures, and the likethat perform particular functions or implement particular abstract datatypes. The order in which the operations are described is not intendedto be construed as a limitation, and any number of the described blockscan be combined in any order and/or in parallel to implement theprocesses. In some embodiments, one or more blocks of the process can beomitted entirely. Moreover, the methods can be combined in whole or inpart with each other or with other methods.

The foregoing is merely illustrative of the principles of thisdisclosure and various modifications can be made by those skilled in theart without departing from the scope of this disclosure. The abovedescribed examples are presented for purposes of illustration and not oflimitation. The present disclosure also can take many forms other thanthose explicitly described herein. Accordingly, it is emphasized thatthis disclosure is not limited to the explicitly disclosed methods,systems, and apparatuses, but is intended to include variations to andmodifications thereof, which are within the spirit of the followingclaims.

As a further example, variations of apparatus or process limitations(e.g., dimensions, configurations, components, process step order, etc.)can be made to further optimize the provided structures, devices andmethods, as shown and described herein. In any event, the structures anddevices, as well as the associated methods, described herein have manyapplications. Therefore, the disclosed subject matter should not belimited to any single example described herein, but rather should beconstrued in breadth and scope in accordance with the appended claims.

1. (canceled)
 2. A method comprising: sending, by at least one servercomputing device and to a user device, multimedia content, wherein themultimedia content includes a representation of an item and a firstindication that the item can be purchased; receiving, by the at leastone server computing device, a second indication that one or moretrigger words were spoken at a first time during playback of themultimedia content by the user device; based at least in part onreceiving the second indication, causing, by the at least one servercomputing device, generating a recording from the first time to a secondtime; receiving, by the at least one server computing device, therecording; based at least in part on the recording, determining, by theat least one server computing device, a purchase intent to purchase theitem; and causing display, by the at least one server computing deviceand via the user device, of a graphical user interface comprising aselectable icon to initiate payment for the item.
 3. The method as claim2 recites, wherein the second time is at least one of an end of theplayback or a predetermined amount of time after the first time.
 4. Themethod as claim 2 recites, further comprising: based at least in part onthe recording, determining, by the at least one server computing device,and that the purchase intent comprises a purchase intent to purchase aparticular variant of the item.
 5. The method as claim 2 recites,further comprising: at least one of counting or filtering, by the atleast one server computing device, the one or more trigger words; andbased at least in part on the at least one of counting or filtering theone or more trigger words, determining, by the at least one servercomputing device, that the purchase intent comprises a purchase intentto purchase a particular variant of the item.
 6. The method as claim 2recites, wherein the one or more trigger words are predefined by amerchant or at least one user of a plurality of users associated withthe user device, and wherein the one or more trigger words are stored inrespective user profiles of the plurality of users.
 7. The method asclaim 2 recites, wherein the one or more trigger words are associatedwith another platform.
 8. The method as claim 2 recites, whereindetermining the purchase intent comprises determining, by applyingspeech recognition techniques to data associated with the recording,that the purchase intent is the purchase intent is a particular user ofa plurality of users associated with the user device.
 9. The method asclaim 8 recites, wherein the graphical user interface is customizedbased at least in part on a user profile of the particular user.
 10. Themethod as claim 2 recites, wherein the first indication comprises anindication that audio input can be accepted, and wherein the indicationis displayed at a point in the playback when the representation of theitem is being displayed.
 11. The method as claim 2 recites, furthercomprising: generating, by the at least one server computing device, thefirst indication; and embedding, by the at least one server computingdevice, the first indication in the multimedia content at a point in theplayback when the representation of the item is being displayed.
 12. Asystem comprising: one or more processors; and one or morenon-transitory computer-readable media storing instructions executableby the one or more processors, wherein the instructions cause the one ormore processors to perform acts comprising: sending, by at least oneserver computing device and to a user device, multimedia content,wherein the multimedia content includes a representation of an item anda first indication that the item can be purchased; receiving, by the atleast one server computing device, a second indication that one or moretrigger words were spoken at a first time during playback of themultimedia content by the user device; based at least in part onreceiving the second indication, causing, by the at least one servercomputing device, generating a recording from the first time to a secondtime; receiving, by the at least one server computing device, therecording; based at least in part on the recording, determining, by theat least one server computing device, a purchase intent to purchase theitem; and causing display, by the at least one server computing deviceand via the user device, of a graphical user interface comprising aselectable icon to initiate payment for the item.
 13. The system asclaim 12 recites, wherein the second time is at least one of an end ofthe playback or a predetermined amount of time after the first time. 14.The system as claim 12 recites, the acts further comprising: based atleast in part on the recording, determining, by the at least one servercomputing device, and that the purchase intent comprises a purchaseintent to purchase a particular variant of the item.
 15. The system asclaim 12 recites, wherein the first indication comprises an indicationthat audio input can be accepted, and wherein the indication isdisplayed at a point in the playback when the representation of the itemis being displayed.
 16. The system as claim 12 recites, the acts furthercomprising: generating, by the at least one server computing device, thefirst indication; and embedding, by the at least one server computingdevice, the first indication in the multimedia content at a point in theplayback when the representation of the item is being displayed.
 17. Oneor more non-transitory computer-readable media storing instructionsexecutable by one or more processors that, when executed by the one ormore processors, cause the one or more processors to perform actscomprising: sending, by at least one server computing device and to auser device, multimedia content, wherein the multimedia content includesa representation of an item and a first indication that the item can bepurchased; receiving, by the at least one server computing device, asecond indication that one or more trigger words were spoken at a firsttime during playback of the multimedia content by the user device; basedat least in part on receiving the second indication, causing, by the atleast one server computing device, generating a recording from the firsttime to a second time; receiving, by the at least one server computingdevice, the recording; based at least in part on the recording,determining, by the at least one server computing device, a purchaseintent to purchase the item; and causing display, by the at least oneserver computing device and via the user device, of a graphical userinterface comprising a selectable icon to initiate payment for the item.18. The one or more non-transitory computer-readable media as claim 17recites, wherein the second time is at least one of an end of theplayback or a predetermined amount of time after the first time.
 19. Theone or more non-transitory computer-readable media as claim 17 recites,the acts further comprising: based at least in part on the recording,determining, by the at least one server computing device, and that thepurchase intent comprises a purchase intent to purchase a particularvariant of the item.
 20. The one or more non-transitorycomputer-readable media as claim 17 recites, wherein the firstindication comprises an indication that audio input can be accepted, andwherein the indication is displayed at a point in the playback when therepresentation of the item is being displayed.
 21. The one or more nontransitory computer-readable media as claim 17 recites, the acts furthercomprising: generating, by the at least one server computing device, thefirst indication; and embedding, by the at least one server computingdevice, the first indication in the multimedia content at a point in theplayback when the representation of the item is being displayed.